search

rancher / rke2

https://docs.rke2.io/
Apache License 2.0
1.57k stars 268 forks source link

Error: failed to create containerd container: failed to stat parent: stat /var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/214/fs: no such file or directory #4961

Closed e-minguez closed 1 year ago

e-minguez commented 1 year ago

Environmental Info: RKE2 Version: rke2 version v1.26.9+rke2r1 (368ba42666c9664d58bd0a9f7d3d13cd38f6267d) go version go1.20.8 X:boringcrypto

Node(s) CPU architecture, OS, and Version: 

host3rke2:/opt/rke2/bin # uname -a
Linux host3rke2 5.14.21-150500.55.28-default #1 SMP PREEMPT_DYNAMIC Fri Sep 22 10:04:29 UTC 2023 (c11336f) x86_64 x86_64 x86_64 GNU/Linux
host3rke2:/opt/rke2/bin # cat /etc/os-release
NAME="SLE Micro"
VERSION="5.5"
VERSION_ID="5.5"
PRETTY_NAME="SUSE Linux Enterprise Micro 5.5"
ID="sle-micro"
ID_LIKE="suse"
ANSI_COLOR="0;32"
CPE_NAME="cpe:/o:suse:sle-micro:5.5"

Cluster Configuration:

NAME        STATUS   ROLES                       AGE   VERSION          INTERNAL-IP       EXTERNAL-IP   OS-IMAGE                          KERNEL-VERSION                 CONTAINER-RUNTIME
host1rke2   Ready    control-plane,etcd,master   27h   v1.26.9+rke2r1   192.168.122.135   <none>        SUSE Linux Enterprise Micro 5.5   5.14.21-150500.55.28-default   containerd://1.7.3-k3s1
host2rke2   Ready    control-plane,etcd,master   27h   v1.26.9+rke2r1   192.168.122.29    <none>        SUSE Linux Enterprise Micro 5.5   5.14.21-150500.55.28-default   containerd://1.7.3-k3s1
host3rke2   Ready    control-plane,etcd,master   27h   v1.26.9+rke2r1   192.168.122.46    <none>        SUSE Linux Enterprise Micro 5.5   5.14.21-150500.55.28-default   containerd://1.7.3-k3s1

Describe the bug:

RKE2 1.26.9 on SLE Micro 5.5... after force killing a node a few times (on purpose, via sysctl kernel.panic=60; echo 1 > /proc/sys/kernel/sysrq; sleep 1; echo c > /proc/sysrq-trigger & ) I cannot spin up some new containers on those hosts (after they are Ready again)...

2m12s                   Normal    SuccessfulCreate   DaemonSet/alpine     Created pod: alpine-nsp4g
2m12s                   Normal    Scheduled          Pod/alpine-nsp4g     Successfully assigned default/alpine-nsp4g to host3rke2
2m12s                   Normal    AddedInterface     Pod/alpine-nsp4g     Add eth0 [10.42.97.226/32] from k8s-pod-network
2m11s                   Normal    Pulled             Pod/alpine-nsp4g     Successfully pulled image "alpine" in 591.95612ms (591.965354ms including waiting)
2m10s                   Normal    Pulled             Pod/alpine-nsp4g     Successfully pulled image "alpine" in 595.01146ms (595.031369ms including waiting)
117s                    Normal    Pulled             Pod/alpine-nsp4g     Successfully pulled image "alpine" in 570.322418ms (570.34343ms including waiting)
102s                    Normal    Pulled             Pod/alpine-nsp4g     Successfully pulled image "alpine" in 592.606983ms (592.637839ms including waiting)
88s                     Normal    Pulled             Pod/alpine-nsp4g     Successfully pulled image "alpine" in 576.327177ms (576.338634ms including waiting)
74s                     Normal    Pulled             Pod/alpine-nsp4g     Successfully pulled image "alpine" in 580.099409ms (580.108553ms including waiting)
62s                     Normal    Pulled             Pod/alpine-nsp4g     Successfully pulled image "alpine" in 599.219733ms (599.229537ms including waiting)
46s                     Normal    Pulled             Pod/alpine-nsp4g     Successfully pulled image "alpine" in 1.576348801s (1.576356723s including waiting)
46s (x8 over 2m11s)     Warning   Failed             Pod/alpine-nsp4g     Error: failed to create containerd container: failed to stat parent: stat /var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/214/fs: no such file or directory
31s (x9 over 2m12s)     Normal    Pulling            Pod/alpine-nsp4g     Pulling image "alpine"

however, if trying to run nginx it works:

❯ k events | grep nginx-wp6bx
102s                    Normal    SuccessfulCreate   DaemonSet/nginx      Created pod: nginx-wp6bx
102s                    Normal    AddedInterface     Pod/nginx-wp6bx      Add eth0 [10.42.97.227/32] from k8s-pod-network
102s                    Normal    Scheduled          Pod/nginx-wp6bx      Successfully assigned default/nginx-wp6bx to host3rke2
101s                    Normal    Pulling            Pod/nginx-wp6bx      Pulling image "nginx"
101s                    Normal    Pulled             Pod/nginx-wp6bx      Successfully pulled image "nginx" in 576.929107ms (576.946274ms including waiting)
101s                    Normal    Created            Pod/nginx-wp6bx      Created container nginx
101s                    Normal    Started            Pod/nginx-wp6bx      Started container nginx

the alpine container however works on the nodes that weren't force killed...

❯ k get pods -o wide | grep alpine
alpine-896qs     1/1     Running                0          7m16s   10.42.204.84      host2rke2   <none>           <none>
alpine-9kf4s     0/1     CreateContainerError   0          7m16s   10.42.49.118      host1rke2   <none>           <none>
alpine-czz7s     0/1     CreateContainerError   0          7m16s   10.42.97.230      host3rke2   <none>           <none>

and just in case, the sysctl commands were performed in a 'special' alpine container (via kubectl node-shell) as:

kubectl node-shell ${HOSTTOKILL} -- sh -c 'sysctl kernel.panic=60; echo 1 > /proc/sys/kernel/sysrq; sleep 1; echo c > /proc/sysrq-trigger &'

is the alpine image somehow 'corrupted'? how do I clean it? shouldn't be automatically 'cleaned'?

Steps To Reproduce:

The config.yaml file looks like:

# RKE2 server url to register to
server: https://192.168.122.5.sslip.io:9345
# RKE2 token used to register to the cluster
token: foobar
# kubeconfig file permissions
write-kubeconfig-mode: 0644
# CNI
cni:
  - multus
  - calico
# SAN for the certificate
tls-san:
  - 192.168.122.5.sslip.io
  - 192.168.122.5
node-label:
  - "foo=bar"
  - "yummy=true"

Expected behavior: The alpine container image can be spin up.

Actual behavior: The alpine container image doesn't work on the nodes being killed after they are recovered.

Additional context / logs: I've tried to delete the container image just in case but it still fails:

ctr -n k8s.io -a /run/k3s/containerd/containerd.sock images rm docker.io/library/alpine:latest
e-minguez commented 1 year ago

Some things I've tried:

# ctr -n k8s.io -a /run/k3s/containerd/containerd.sock images ls | grep -i alpine
docker.io/library/alpine:latest                                                                                                                application/vnd.docker.distribution.manifest.list.v2+json sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978 3.2 MiB   linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x                io.cri-containerd.image=managed
docker.io/library/alpine@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978                                               application/vnd.docker.distribution.manifest.list.v2+json sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978 3.2 MiB   linux/386,linux/amd64,linux/arm/v6,linux/arm/v7,linux/arm64/v8,linux/ppc64le,linux/s390x                io.cri-containerd.image=managed

# ctr -n k8s.io -a /run/k3s/containerd/containerd.sock images rm docker.io/library/alpine@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978
docker.io/library/alpine@sha256:eece025e432126ce23f223450a0326fbebde39cdf496a85d8c016293fc851978

# ctr -n k8s.io -a /run/k3s/containerd/containerd.sock images rm docker.io/library/alpine:latest
docker.io/library/alpine:latest

# ctr -n k8s.io -a /run/k3s/containerd/containerd.sock images ls | grep -i alpine
#

# systemctl restart rke2-server

# ctr -n k8s.io -a /run/k3s/containerd/containerd.sock snapshot list
KEY                                                                     PARENT                                                                  KIND
005debb0c2c4241d0895465eb368232a8f81bfbe237d5557f1b8049b95c5bf1d        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
027ccde8fedcc21068157135f868a9b9abb43a2af2982c681fa5e9b0930eac7d        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
091ab7ddf0f80b7d11256e2d67a6f2d20dcc406c3006fc07f575b8d206f2bda0        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
0961a010b1763cb91af1e37d8cbe67edd3cc11b064cf83ef7ad0a1eb3790101b        sha256:29fa685bbd7029bff01b2685945dad43a3b93cfef4494a66d732d371c3209eba Active
0a3fb44968b4cd2847584001dac6bb7d63b8ab31b43a39586229d5eb6908f4e8        sha256:0a8259ffc5ffde0dca4c106d5d43c70d1cac990567ab930ee49f335e6ffa2c98 Active
0f00afc24d6aa93e48d48b84e7edb8a6cd7bff5f831a13357d988995af5740e0        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
1162340cccbb7111ed39b180bd1b89ea930687f45059b5b3715cf97b0b2582da        sha256:d13700a9546f2f7afcbe0fe33bd519097dfd723cfc3f6f1e35880d067a52fce3 Active
118454793e17da0b4a8d9e4bcfd0711686cee424fc67a9fe1e19aecd0db49de9        sha256:a726f89c78ec82f4307e818c5887d29d2bbc89cc8c07e92862e69d1aac5f3ddd Active
12b7dd84ef85bebf28290de669866ad1ad39d3e73fc7e1f66fc1aabc68e7ae1d        sha256:97124a0da9e9a24a5425acf487afe40dee074582b494be52d872fcfb1172df7e Active
142ecd92af022d7a403906fb3265b8e4b98e46712af372b95d8cb0d215ff580b        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
1e174912b30322a64daafbdcc4c8a36561b16708ef33d6d85a25eabf906f6e30        sha256:bfb33b7e28ba1496e1f14c350aea3c8aa956fa5d96c43de3c886ba10c3750339 Active
1e3b7f3067a07e5313b33660adbf0d5787b071e5e62234f5ac4f182f04ca50d5        sha256:97124a0da9e9a24a5425acf487afe40dee074582b494be52d872fcfb1172df7e Active
1e6e8027a02c9845607eded0278b43f4180fd5e908665cf82ba23801a93a6c84        sha256:f3e90bc2dc75fb4104e8e894771b2e35a22b2e3e9b0e716e7cfd2faf3687d3bb Active
1ff12d6665316049a71d0209f6c3560f47326f77792ba60f7887a3c85ea05fef        sha256:bfb33b7e28ba1496e1f14c350aea3c8aa956fa5d96c43de3c886ba10c3750339 Active
20a87314ea5bdf98ed3593889205af30b06ee4216e44cd1d64d1a66efbb1a2c7        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
23b1fc553d244e21d7f2884e16f7401d54af21f53c8e3b7d9490feca9b903f1f        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
24348e4933b2a54406a6edb08f79e8a8bf8a46f9bf84a867ae3ad52d5e3d120a        sha256:86c54f145a15ed72508a406a556f5a278417280664ca92e62d06a567371c1b1e Active
296bfe5c0265e13d3575dba5e6cd29d717357e7ff0492e3ebe29f58b034d386e        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
3141fe4066ee0b09a55427fabaf736fb8ee1abc7dc5c7bedc3dec00d2575ca04        sha256:90511483e3e957407eca28607a16561a601ad73fd42052fe1de142456f3d8d08 Active
34dc1de91d4016a8f61406bc947d10c97f4c5a2085ff8b21e46979f411eaaf1f        sha256:cf1ac4276aa1377bd7dfb060bf8beea204a677cb86bb5aaf5f81c1e04aae1f8d Active
39cf83ffee109e60cf093a6defc795231b39dc4233afa99149a5d57b729df5af        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
3a14a83c679b78a8262e56a55889636659e749a1b39195f681927d7767d1e90d        sha256:0a8259ffc5ffde0dca4c106d5d43c70d1cac990567ab930ee49f335e6ffa2c98 Active
3b7df77c08eaa88d9da57d733af0c9f54a7fd9ebcbb72d4851f7f1e1a9783ebe        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
3bea558caaaf45f70323329f2f45eb3c7d961d0e64792feb3cce8bd12489f681        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
3db50fbc13945a942122f593643c6d74cc302411237d6bbdafd9fd419b5e86a5        sha256:cf1ac4276aa1377bd7dfb060bf8beea204a677cb86bb5aaf5f81c1e04aae1f8d Active
3dc0be78ec8755ec72e293984595a39da0cbb325a46488022a30dad6cb88507f        sha256:119ce9fa08fd73bfd1727dff4d58f01de701363b89559a5eb59c00bb7957e26c Active
3dcd77e805d742694cd7bc2ac251e637fba384f50fd54f0bc82556755f27621f        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
3e1cea634f56481c2212ec61289d1352d585981c3717c0c7cdf522c864d1fd4a        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
4229be441a8e3419baad32bf50919d66e5df63cb4833272a81aec8650cb7740c        sha256:786d4c214142c3902b3066b9960908acb8771205102cc5f3df3c09d13d297dc2 Active
43232627695438e1a7ce6fd19d0ddfb8940a593151c9b462145affc1b17c0db8        sha256:a1ea3fbe4d8a234d1a7f11c4717566efc9f9b8aa993cbe1f63242edbedf094ff Active
4393c6b83d96ed115212794d85c36c5aa37dd5ceedd37875f571ecd33d058791        sha256:119ce9fa08fd73bfd1727dff4d58f01de701363b89559a5eb59c00bb7957e26c Active
43dc2641ef85cc6da8fbd9282c1d09aa2d7ba4713843782bcdb678bbf5a7c7b8        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
4652167729d189c20a27e023649b2968a783696bf9a07d579f2381ff28ddd6e8        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
49032cad42f71f171397daa96a801e87055b7d2f28babc7e907c948bd2125de9        sha256:119ce9fa08fd73bfd1727dff4d58f01de701363b89559a5eb59c00bb7957e26c Active
4d16aac0519f226a0df324b8c5fa0fc348b20369d77c72574c641622908aa582        sha256:29fa685bbd7029bff01b2685945dad43a3b93cfef4494a66d732d371c3209eba Active
52f6c6e552f39433dda2a4803deccc5bd13a2ff34f0c87c4ac677f94aa204a72        sha256:d13700a9546f2f7afcbe0fe33bd519097dfd723cfc3f6f1e35880d067a52fce3 Active
533862765625cde4a0ec6715351cbfd25f4e8399f9a9e8db474f1cde546535a4        sha256:119ce9fa08fd73bfd1727dff4d58f01de701363b89559a5eb59c00bb7957e26c Active
53aa311e86d8a41dd83338b1d7b9c45975d64971c2319a39b0fe7b0519bcdba6        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
54d96f1352d17a8be64136764380c13d136e7f33f65796e332dd1b411034c5ba        sha256:0108616b116dc3b4a23ed1ee2196ecd49a903a6a123d3ad42df731eaa1e5f417 Active
56b69e5587f7a5fa31df0cc29c680ac7e21963130a874fbb00c4e6f0f1940dd6        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
5702b04be8dea55cce3683c800ec89393d503ecf827ec9c061c05af518454d52        sha256:119ce9fa08fd73bfd1727dff4d58f01de701363b89559a5eb59c00bb7957e26c Active
5f5584a57c714338e4576a0ad058c837ac2cdb086d11798ffe2a4a6ce67c73bd        sha256:07e522434f3f44bad541f3d3c8f9ea81f8090d1d3b09ef3f8367ae4bbe94a035 Active
66d33696ce7f3adf94a0b6fec87ac218cae44097ee9934f9daa9be00cb2a1ef1        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
670b6041f8badc1b7c5c54a1374086dc890c1eafcb143c2546c8d2d16659d302        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
68979da3dfdb0f2bd0681231a352a9918af29707d55dce1f0d0535b9af9b7733        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
6b5cb1930ca6812fb10fcd1cd952e9bddb64fc8e4544a567cc15524d02f57e8b        sha256:588f1158dfe9f3b3dcad8eff09a1037e035dd95c97257e468158bbf869b1c980 Active
6c3378f4e5804a29a04614fade59a700836b1692e94026f3a3230136c0ea17db        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
6db1edfa66974e4fecc9f52303149ec6d5c855497d0d0f1ec63e96d8032f8417        sha256:159a7de360f137d456cfb4a27da4dd348fafec4b324dfdd925d113c4fe73f2e4 Active
6e2a665c0b922f6c9b418948320a3afaf61179548c09071428b1130f0df0497a        sha256:b64283fc8b3b2640d7b72a94ae49e45e287befd9b59b76d62803851448a6c587 Active
7225220e0f4923d00a63bdc1eacb8f891c2b9818a544ee438e8301bb20728b3c        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
7376b5c08697b37ae4222a0a9757f215408088a0be7e2609d71b11caf2c46d47        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
76690b1753e76b1fbe9501fca1220b60c8719044e90e3ba50117acc227d36b68        sha256:119ce9fa08fd73bfd1727dff4d58f01de701363b89559a5eb59c00bb7957e26c Active
7ae1f72fffd3e4bdc3ef288b7cad3c398d7760e3fa4b7e385ef2426b36559e1f        sha256:29fa685bbd7029bff01b2685945dad43a3b93cfef4494a66d732d371c3209eba Active
7e9966dc10f94cee052bf15bb1926a92ba930c98410c38606237112565d02404        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
8085942988f4bd4d8830ebc5de8b50e63dc3d8145e18c37e77ccd84a80f2bf92        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
81aed34f2855d1ff49c76fb329519f1d2e2df200cecbdc274d4fc1ec90e6d31c        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
835bcf958964a06b9e0e2df618eced2a0820ff33537e47a368e5bc8a6c16d3e0        sha256:588f1158dfe9f3b3dcad8eff09a1037e035dd95c97257e468158bbf869b1c980 Active
85e21d5b3fe86d0273765550e859ca48bc27beb834dac01fef8f038c40230cb1        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
8607542e686d4e98178dcd31721309f08a1b22376d655b1f77cddd568f7eecf5        sha256:cc4fb2df65e7411567ec46817dd2a2d2d121b31680fafcf83a0e305c68cfd46d Active
87d75994b125a78bf31d2d0362dacfb28c4e6080cace0436faf8e8c582ee4e31        sha256:119ce9fa08fd73bfd1727dff4d58f01de701363b89559a5eb59c00bb7957e26c Active
8ca447a19ee2ac0a1e6d39517dda0359658798cab5af66cfb08f7950ff0ec6b3        sha256:58282526c353536fa1fe34f70905f96263c82431c42607474794831cdd5aa88f Active
94cefca4f37dadc54a62615dc093561b8f8d266ed6fb6717ebc730a60b375465        sha256:178132d5bc4553b0f35efac8626e2f958d2e1bf25ce6581e4dbf15eb7ce20550 Active
95080feb9b22f36816ee5706cafa902c781942b165d86e2b03230a1e05a56a92        sha256:97124a0da9e9a24a5425acf487afe40dee074582b494be52d872fcfb1172df7e Active
9948e348a71f0b04e03854e379da49222ef64ead6d82520dd8ad298efa23e039        sha256:cf1ac4276aa1377bd7dfb060bf8beea204a677cb86bb5aaf5f81c1e04aae1f8d Active
9c7fde0183bd40364da2402b6fc77b57c0d4678c63c4d2bc29f4a80e49a17ca1        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
9d7de43c6c51c2684f27d6ce53536515dbd067a98a14fc9e5c389f9baa89fc12        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
a0c5325f8c242c30419fd7c6c219e1c2e986c3ccda8b97987b340b2f063c9b04        sha256:07e522434f3f44bad541f3d3c8f9ea81f8090d1d3b09ef3f8367ae4bbe94a035 Active
a1de81d99f12a72cfc0f31895e7dad6eda334d8c0af7ab6d099c9df338a1649e        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
a97a30fb754eeea5fa162b6e5fd7dbc8a0211fb90283d36a16c4133e6ab80f95        sha256:8c7ced29af94f2797569988cf900c5b75296476e5c2116e5ea8eb2e1801b0f5e Active
aaaa1b6c9fc10002a91de0bec9be21f3486bcdd350c79e37ea56a5f0197c3271        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
aaf911300918ada9b354c2729a1d9d2b3b6489f8e643dc99d116237dce03c04b        sha256:97124a0da9e9a24a5425acf487afe40dee074582b494be52d872fcfb1172df7e Active
abef04069cc0eb0063a2a5017a09856918ee2459c6838dfe6491e9714cb8b427        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
ac286e3b48441c111d325b9970483d0c83a59e5e8903a8461e77b9de632fe97a        sha256:119ce9fa08fd73bfd1727dff4d58f01de701363b89559a5eb59c00bb7957e26c Active
acc68c7b7f87d8d6fc2b7ead9d0310b0391746321be13ab3624406d691b6a788        sha256:58282526c353536fa1fe34f70905f96263c82431c42607474794831cdd5aa88f Active
ae7fd1ad0289350358ffe6a3fdd1446b9867315af84e2534774d7edb7778e657        sha256:cf1ac4276aa1377bd7dfb060bf8beea204a677cb86bb5aaf5f81c1e04aae1f8d Active
b0c4eb70dfdbe04cf30cdf2aa302d111a867420fb5cd9b78342ce7e37492facb        sha256:29fa685bbd7029bff01b2685945dad43a3b93cfef4494a66d732d371c3209eba Active
b4279564ec09cb8b78299189f10f38d82841324b10df19f2a320e713148ff96e        sha256:04ff82f8dc1cae486c38026ed83c2f4b0d83479310f32c783bf655b1d219aa76 Active
b82ba93b8aca1a4f8d6a98283c7c5c3ba2ee0d8b5e658147e6c601a2305c2f79        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
bee40de35ff7b8b7085c43e110e2094bee2d4524e1639693290c726b30ff268a        sha256:f3e90bc2dc75fb4104e8e894771b2e35a22b2e3e9b0e716e7cfd2faf3687d3bb Active
c44c0367714e285ad0da9751e1ef60884f7462e53aee71bef33346479f352cd7        sha256:cf1ac4276aa1377bd7dfb060bf8beea204a677cb86bb5aaf5f81c1e04aae1f8d Active
c63967e6a99472d1f4080668117d1f751e96b85cd49faa70170ec9de8804d710        sha256:a726f89c78ec82f4307e818c5887d29d2bbc89cc8c07e92862e69d1aac5f3ddd Active
cd430f30c6e8ec9a6bd0e7da7d85f8430be30d2120312e6505eff52886f7d9a7        sha256:04ff82f8dc1cae486c38026ed83c2f4b0d83479310f32c783bf655b1d219aa76 Active
cdc4679fe5b334d50a5a2d7375b6332898d96d781e22656f9ba2eccdc6c2c58c        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
ceaf8975e96f7460417d1547607a517a40e5834470bc0450394e59f4bb742b4b        sha256:cf1ac4276aa1377bd7dfb060bf8beea204a677cb86bb5aaf5f81c1e04aae1f8d Active
d0cc7796ee28376c84d3965a3ddc92d3c77a8246f3b2b98969e5ee91e136fd9e        sha256:8c7ced29af94f2797569988cf900c5b75296476e5c2116e5ea8eb2e1801b0f5e Active
d2016010bdc2783ded3361b2fa73a3a5e270fabc8bc83f4ee0ab17b444e25c4d        sha256:0108616b116dc3b4a23ed1ee2196ecd49a903a6a123d3ad42df731eaa1e5f417 Active
d6077a08ee9a37ae796fc7d482c6927fdc79e4b3e1b8185b09ad41ae8b4d15bd        sha256:86c54f145a15ed72508a406a556f5a278417280664ca92e62d06a567371c1b1e Active
d6426b0b1b079d7664130706100f9b243ffeb41867c76e1d5da48af9d51eda7e        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
dfe9ab1b4e5cc52db398be714e4b8fb6aa7b1a1d41d848ad5df4b2ab46ec1500        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
e66c3db073f49ffd7ae3420b7a2ed91c7d7cee4d979fb55af9d8bcb2ec2954d2        sha256:159a7de360f137d456cfb4a27da4dd348fafec4b324dfdd925d113c4fe73f2e4 Active
e69ed9969a90b5e712cc7c3b97ea409b64789dfb300aaf7ad159223ac1bc3745        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
ef7cb6aac650b519daa8c77ffe3bdbdb0985d97438a9017f112ad1af95ba73de        sha256:cc4fb2df65e7411567ec46817dd2a2d2d121b31680fafcf83a0e305c68cfd46d Active
f31aaf7b2aa49497e8db7c89da99bbb9b54af845abf9d4ae7a15ff4a97830446        sha256:e916069494b485cb0ac31864cad59236e24ca497399e0a9d50ac5742ce012b28 Active
f46a32c40989d63151ac28a4eda6862d4a6867b3af48f3f2d160122f5fb54a15        sha256:a1ea3fbe4d8a234d1a7f11c4717566efc9f9b8aa993cbe1f63242edbedf094ff Active
f53518f0a16b3ce476352ce4dd1143359a673335320cfa01b304710b2bc7ed77        sha256:cf1ac4276aa1377bd7dfb060bf8beea204a677cb86bb5aaf5f81c1e04aae1f8d Active
f5613724004c4056ccf3406ec32e0e07d7b5d1a1e38c9d3f33571758c36fc7f2        sha256:b64283fc8b3b2640d7b72a94ae49e45e287befd9b59b76d62803851448a6c587 Active
f5893a0022f5da7f2c6737f785267abdd6f0f5fe1bd08e9f6dac95358849fbf6        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
f605739eff563522beed302171a5b93e498bffcd46bd970dbf74b89fbdcd25de        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
feb77b26d063c2318f76e141dce20d1abe302111314c2c2c041f3cbde72e113e        sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68 Active
sha256:0108616b116dc3b4a23ed1ee2196ecd49a903a6a123d3ad42df731eaa1e5f417 sha256:cb60fb9b862c6a89f92e484bc3b72bbc0352b41166df5c4a68bfb52f52504a7d Committed
sha256:04f0b72a99243f128a871acdc2d015cbbd37a1ae9119e0e6a754c29ddd86f5f8 sha256:54c714144fe19aa4655b686cdb12d686a38e36d5d5a0d290548beb1810146135 Committed
sha256:04ff82f8dc1cae486c38026ed83c2f4b0d83479310f32c783bf655b1d219aa76 sha256:c124b2c4d0c2ffb1abf039e47ad041f55a876ed1e0060f58e6e0ac8e0588e782 Committed
sha256:05b8faf1622fb061cb95828a6fef9b659382505d81c5622b1c258f5003b27e64 sha256:de369c4e2ef0226507f9689d776a96f2d5005ff5f322b0501e6da8614a94ef53 Committed
sha256:0680945c9633c26abff2951321d5308b01013b3f40e4d5370bc707e57a4d329e sha256:d9d63d5691f24fa635706bd0803dc116d31f57870cb41471ec62b7840e487132 Committed
sha256:07d1df5250fa19cbba9e857aabcc302f323f33cb6eda5bbecf4645f1dc704347                                                                         Committed
sha256:07e522434f3f44bad541f3d3c8f9ea81f8090d1d3b09ef3f8367ae4bbe94a035 sha256:cfc0039532c6d251c6eb90fe887f1629209948fd4b39ec1d218254f4da87f88e Committed
sha256:0a8259ffc5ffde0dca4c106d5d43c70d1cac990567ab930ee49f335e6ffa2c98 sha256:f7cdaa8664dcb7281f3d71f98a3bd29296a11ee542e12f380d509ec2b196f22e Committed
sha256:0e04e159318a93fa02c1615e565f4ad8ae6a28f7c3b306ebc78366f01d82b7f9 sha256:9aaf79c2f1043ddcbd161813aeb9794b08041d410003d418fca2c6e08ae70230 Committed
sha256:0ee3289e65e8a1bda84dcb70f208c45732775ce2c754a4d5bc6bbf684507854e sha256:7b0fa559d15e5d9ffaaecd54a9150240ce4cbdfe3630ebf0275ffa9af4496df1 Committed
sha256:1021ef88c7974bfff89c5a0ec4fd3160daac6c48a075f74cff721f85dd104e68                                                                         Committed
sha256:119ce9fa08fd73bfd1727dff4d58f01de701363b89559a5eb59c00bb7957e26c sha256:c2174a35d0a3a1324f91b3d29bcaf3270c966ae6d0f82c51f048c1511ee0561f Committed
sha256:150e4c2a0f8e85d47e8490899e32a908aeadf5d6e6f615fa8fbb1d95c822f115 sha256:b8558647455474d3f08d3fbc59beda69b2c4dfb2d9080c8f9e8593f167b5f6a1 Committed
sha256:159a7de360f137d456cfb4a27da4dd348fafec4b324dfdd925d113c4fe73f2e4 sha256:5ed6bb5a36424feb61d1a2e9fa1805e9bf3be47453e9700d5d46e623b68ed663 Committed
sha256:178132d5bc4553b0f35efac8626e2f958d2e1bf25ce6581e4dbf15eb7ce20550 sha256:be8757e8c9a30bcf602c202290b903820fa890a108bf1c03f06e11db2e3cf9e6 Committed
sha256:179a82b4b6d22589fe605fd3a8b301a09119ea90b826c6a5fd7718a4946218ed sha256:d058b2e299906a3d6814ee8759dc2dc533f1315ded793156909e3a0226b66ef2 Committed
sha256:1816b35ece935f0e31644e8bf088cd49df1d9b6aa710277e5584f209761e087c sha256:914d547a4f225fd23ce28acae403d224efb1c7a731460e45660b5fe27f11dce4 Committed
sha256:1ce51ff80e698669a6891a13d3fc33b855adf85e604817deb4136f44482904cb sha256:7dfc0babee3e92f7bd4ab37d9260440b0166397ca70896f8242100cf19b54cf3 Committed
sha256:1d0506f41c89884eca81d92f80479ba493be75903ca1bbc12d25e8883d8cd65a sha256:fef9a9ae4bfd07f361144f7acd9c2f768c9670a2eee895f9555f5bd5f960eb6e Committed
sha256:1e2d7a9c800c98052f5fa26d6c059d0762ace6d6e83624b9a3dc3e9063430506 sha256:dbdd730772922b9fcfc5b88c95617c21ae0b3d9beb274e9db234dfd2930a1d1a Committed
sha256:233d20966e0bc0d71fcbbb4e0dc683040a38f1cbeb53910ba370f6289679d1d3 sha256:ebdee40a13a8a535d59ad25ba2f62e4d436c4d7081e80a44e5b884474f795580 Committed
sha256:269056aea1547ccb85a7e4b5aa0cfe362fbed722f14df4586fa1dcaba0dddb12 sha256:179a82b4b6d22589fe605fd3a8b301a09119ea90b826c6a5fd7718a4946218ed Committed
sha256:26e009c8dab75b9f2b9e38c6887f031d5ee0b39c15885cee73324f7c638445c7 sha256:6a8e5cbc9a4699caf5df6b7737a7ab74ec4c4267ee8ef9a8a53691eaaf233b86 Committed
sha256:29fa685bbd7029bff01b2685945dad43a3b93cfef4494a66d732d371c3209eba sha256:eaea05884bde4441c0a8a223affa360a8372eedf75bbb4b14025e2a113858c62 Committed
sha256:2c36830d2b3b9931c89fbb58955e6a155bc13b084fbb6a0746c88e0f9e31dbac sha256:9e4efa6e69767843fcf5bbeb87304dec035361c4ffb8484189c03f29a17bba1b Committed
sha256:2f98ae9a4ea7a6984da8eead7cff8e1d49027109bc7884e4261820a1ddea6cdc sha256:9a24f4c2114ddefbec763acebeb61ba30563db97ab554e13aa59cdcf12165707 Committed
sha256:30c4d9fc6bf52615784c3dbcaa808085feb8ae94229022beab70bfab5cc56100                                                                         Committed
sha256:31dbd8690f706b9960dd8882e2bed8c2c0e1e099c2a6f993f2adfd27cf06f2e4 sha256:666b26e28258c9722dfd3614acef2c973bdfa0a66d121ed43156f82eb722a3a0 Committed
sha256:35f446932246e717f6adbec74fe5f1ec0ecced5a9ecf8623cbb1f0c82dc7710e sha256:31dbd8690f706b9960dd8882e2bed8c2c0e1e099c2a6f993f2adfd27cf06f2e4 Committed
sha256:3745a2d825305d88cd6b4aed9b5b2de8eb2a36d237dd471368d18cf5d96e907f sha256:87d5589860cd08d504e32e6e17498af935f95dca535bbb5439ea50049047dfb4 Committed
sha256:383530c0ccf430e191a4d441649090235a8a4b0a23f0e9e1af5afc29cbe89975 sha256:f79c4584d9c2c73bb9cc8d438a8d5c50ea7063df6baacb5d1423b6538a48e0ef Committed
sha256:3b891a13ddc94b26b977c6fe49461294d8a5a4ce731d8d273c24048a67f25c8a                                                                         Committed
sha256:3f49d783f3f39276404439babcdba36ee338634c40a9a5a8a28df522d24d0c30 sha256:f08f0e2f3cd0106f219789426127ab0b99daa24ff236904c0c3b3d72fb8f25ee Committed
sha256:40813db08f1c36bbeb71e194717970112a88e4c33502292c0583014f36278ae4 sha256:e2fc87ba0086e7db714848379d05c58f8ec7a801beed60a14d9370c4c148092c Committed
sha256:493f93de45de060d0dfc7c49f5a409a6b2f743b7fb0deadfb74a40b3a50ef4c1 sha256:2f98ae9a4ea7a6984da8eead7cff8e1d49027109bc7884e4261820a1ddea6cdc Committed
sha256:4c8a20034222d03c096a28cf47459fe45a43d3edeabda95f86f3d830bd4380ee sha256:f5ecd8d7cf73848fa7a5532706439062c198972cfbc757dbf4f3016fccab2172 Committed
sha256:4cfdb156caefdad3a177a48d86a7a5062801869e628ce4c38bc7ddf01b0fa9d1 sha256:e50ea7ba0d96cc84932c911015cbdf69de2f57850666466eaa1a42d532e1bc46 Committed
sha256:515f2f2eb44292a81f96bf82a38285c5ce30cd6bdb9828d0551f528371588060 sha256:de39c9d61b393e5ef354cd247bb2b83cdfff594cb8147b6b029abfa0323c1a22 Committed
sha256:51b740c4b48eba19c400bb3b03a86b879eb1a50305b8f882f4ea8e44d0a8edcf sha256:e5a8a3ef277f4e6738e6c89125857073f570831235bec75e580d289f6d9ad12a Committed
sha256:52d58f02f883a2c3687215b71f7679b48d75f52564c248ade22d8ebfc0514b91                                                                         Committed
sha256:54c714144fe19aa4655b686cdb12d686a38e36d5d5a0d290548beb1810146135 sha256:f4d5b6c9593bbef07f183d0524f07fddff0120eb138042e16c46eb75d2355da2 Committed
sha256:58282526c353536fa1fe34f70905f96263c82431c42607474794831cdd5aa88f sha256:6d6b545613ddaed5ba5aaec5fad03401ff9532d0ee7210158fb541f56ab57575 Committed
sha256:588f1158dfe9f3b3dcad8eff09a1037e035dd95c97257e468158bbf869b1c980 sha256:e998b554f2d9e7d57713d4c78a5baf8da288990f487b37d0671c067c2a7f7b71 Committed
sha256:5a65d21aeee5bc0cdbb39b73f14304c111118c700125f3d97572498e9ee6a7de sha256:802b401a7d15f88e2c20b60a91040577fc12222a255ebeadbf7a62f7905108b3 Committed
sha256:5b4205a7509156aa8e11528ddc09b6f13ec5ba1681be8091b529469fe07f1b0b sha256:b71a12fa31024ac7e5e7f5a532a6665893c3482b470f7a98cf5625d438168cb1 Committed
sha256:5ed6bb5a36424feb61d1a2e9fa1805e9bf3be47453e9700d5d46e623b68ed663                                                                         Committed
sha256:5f40d2c2af6bd6c289f284b2ffb1022db6984edf7005ac791c3723730dd40111 sha256:bd85a340ea268d71b49c16a693f1764267228a5787fb0da2b81dc6c19fa9f97f Committed
sha256:6416129324c14f37fd5dc9409273bf7bad9e5ac0eaebc806b06aacb4036a9459 sha256:d0157aa0c95a4cae128dab97d699b2f303c8bea46914dc4a40722411f50bb40e Committed
sha256:666b26e28258c9722dfd3614acef2c973bdfa0a66d121ed43156f82eb722a3a0 sha256:68f1b23f097b3c8d1f40511600e055d1c3a3427595afa6234b6613e0dd38b224 Committed
sha256:66fc065f4bb48bf1b0a925e17f8bc4183a5fc45a7ef6d997f40c9c7fd6c65bde sha256:51b740c4b48eba19c400bb3b03a86b879eb1a50305b8f882f4ea8e44d0a8edcf Committed
sha256:6838bd6d9585c86b3c9303dbd62c3afa520cabd07bf8576b7adb01ecd334bdbf sha256:e14da2d720e0784a15e2e1b04bb3ed2e1aa537131001dc8886aa57a90136689f Committed
sha256:68f1b23f097b3c8d1f40511600e055d1c3a3427595afa6234b6613e0dd38b224 sha256:3b891a13ddc94b26b977c6fe49461294d8a5a4ce731d8d273c24048a67f25c8a Committed
sha256:6a8e5cbc9a4699caf5df6b7737a7ab74ec4c4267ee8ef9a8a53691eaaf233b86 sha256:bd56d64a4768ae109c1b382a438262fc2df1896a1a4858deaae285cd18f3d1c3 Committed
sha256:6d6b545613ddaed5ba5aaec5fad03401ff9532d0ee7210158fb541f56ab57575 sha256:c5ac479a4d171506ef5273b6c5f823102a3bb8710eec55613694e0dd7a91a5bb Committed
sha256:6e5daea76b043659b80e5d2b756c4d364fd663f683a70b7b2ec0aafb14056413 sha256:4c8a20034222d03c096a28cf47459fe45a43d3edeabda95f86f3d830bd4380ee Committed
sha256:6eb39a4c45053b99d4562e2bf1c6cf2f13b9cfb634447c6cdbf21b55c822ac93 sha256:40813db08f1c36bbeb71e194717970112a88e4c33502292c0583014f36278ae4 Committed
sha256:724c3e4ae4a0d3094154899d85b59c3e776b4a6267103cd4ff715ff17bcd0afd sha256:fd6089bf399876168a847c48fc04d7ca8a29bc0e158808cea991cb5ed79ba532 Committed
sha256:7668a5eb0f0716d07ec2f3b0e12d431405311557ce41e61f881cfdc125ca931c sha256:35f446932246e717f6adbec74fe5f1ec0ecced5a9ecf8623cbb1f0c82dc7710e Committed
sha256:786d4c214142c3902b3066b9960908acb8771205102cc5f3df3c09d13d297dc2 sha256:be8def87eb228e40250344b611afbd18a24264bc6aa1326c878732c7b91f6962 Committed
sha256:79782f884ba606bd680107bcddc8e79e4e55d555942304c022dcf7bdadef23b7 sha256:b41b074d18a7d752bc046459e53d60f6b02f13872396f3e19b4482d341a4345f Committed
sha256:7b0fa559d15e5d9ffaaecd54a9150240ce4cbdfe3630ebf0275ffa9af4496df1 sha256:aed23d493e9662bea9f0b1ecb55f0438f8101a025cad07916277a83ba177571c Committed
sha256:7dfc0babee3e92f7bd4ab37d9260440b0166397ca70896f8242100cf19b54cf3 sha256:3b891a13ddc94b26b977c6fe49461294d8a5a4ce731d8d273c24048a67f25c8a Committed
sha256:802b401a7d15f88e2c20b60a91040577fc12222a255ebeadbf7a62f7905108b3 sha256:0ee3289e65e8a1bda84dcb70f208c45732775ce2c754a4d5bc6bbf684507854e Committed
sha256:84ab39a267f4d29b82795947a07b19ec459f8333a8869f8da4b6fbb15a2cc690 sha256:8b3694a79e598d6c815d528e468c509af7ebbae35e734895d561eee40e699243 Committed
sha256:866454943ef40a2e85443b3c6aa36e6ecc4b11fc197de519ec2d999b1b9f3890                                                                         Committed
sha256:86c54f145a15ed72508a406a556f5a278417280664ca92e62d06a567371c1b1e sha256:515f2f2eb44292a81f96bf82a38285c5ce30cd6bdb9828d0551f528371588060 Committed
sha256:87d5589860cd08d504e32e6e17498af935f95dca535bbb5439ea50049047dfb4 sha256:cb4596cc145400fb1f2aa56d41516b39a366ecdee7bf3f9191116444aacd8c90 Committed
sha256:87f5fe271fde4b62787033b985a5851a06e87a4a9d5a54f6bc6ee8b54db9e4d9 sha256:5b4205a7509156aa8e11528ddc09b6f13ec5ba1681be8091b529469fe07f1b0b Committed
sha256:89fafeeb9db96ca37f3214c0d14ce160ce82041b284ecf3a26641d3b3b5359e5 sha256:ec22d9ca60728c69acca7909d96922dfe5ceb4665ecd6c91b3282f18c3a58424 Committed
sha256:8a8c5bd45d812d57aa49d9f8e61e8cd7dbad31f4eba95ab5246fca8985a38b50 sha256:a10ec15ee5ef56b79fc519c6a56d77fc53a7d1add70e0809f0cd71f01ee4243e Committed
sha256:8aa2c77f6b0573ac718a7468d4d47cbf09963364d0c722906f9fac5e9c89186b sha256:233d20966e0bc0d71fcbbb4e0dc683040a38f1cbeb53910ba370f6289679d1d3 Committed
sha256:8b3694a79e598d6c815d528e468c509af7ebbae35e734895d561eee40e699243 sha256:6416129324c14f37fd5dc9409273bf7bad9e5ac0eaebc806b06aacb4036a9459 Committed
sha256:8c7ced29af94f2797569988cf900c5b75296476e5c2116e5ea8eb2e1801b0f5e sha256:ede59ced70e61f7a85dc7e74d4a3bf7c6aea92274afc15fad005b23845b64430 Committed
sha256:8ca1271bd3b844e4ba07eb1b52fa4d53ee1e1bf39b6d24eaeb88abe42119de0b sha256:acd6dec99d96a1d53173d713b6ee72f13edf346aa67eb357f8efb155697817f2 Committed
sha256:8e3473c30af5cfb0c0a3cc6eb7cae4622a84098987e3ee02a420bcb11e84407b sha256:987cb8ed593a5c025b8e9bbe2cc8b24f93b28e583f8f35ba64d5b25a6ea581d1 Committed
sha256:8ef2f4f683a86b6ee96207519e77296b52847f45f89572cde7a9edc7c1100ca5 sha256:05b8faf1622fb061cb95828a6fef9b659382505d81c5622b1c258f5003b27e64 Committed
sha256:90511483e3e957407eca28607a16561a601ad73fd42052fe1de142456f3d8d08 sha256:ca52dc50a03a62727c4337675c2f3a66c8b0a82c98d53fe375eb7ca2a141a20c Committed
sha256:90aff515acf292d70d9ac429d3cfa134341a8c20f9cf689fae1cbd4ae0916c2b sha256:6838bd6d9585c86b3c9303dbd62c3afa520cabd07bf8576b7adb01ecd334bdbf Committed
sha256:90df11b510893c99ddcd4549452390f8947b084cec1e301270926a4a8f77a1dd sha256:1816b35ece935f0e31644e8bf088cd49df1d9b6aa710277e5584f209761e087c Committed
sha256:914d547a4f225fd23ce28acae403d224efb1c7a731460e45660b5fe27f11dce4 sha256:de1b174a0e25174b60881457db7ba1282ed3705bb24a9f5c8b7766cf2521e491 Committed
sha256:922ea7b2b282174db2272a0e50acb4a6946ed61a108a48f0adcca5463268d929 sha256:0680945c9633c26abff2951321d5308b01013b3f40e4d5370bc707e57a4d329e Committed
sha256:9359f802c19ffa094cc2e41b3e3c608577c0a6ea2e79fc61a172432c1f9be5c7 sha256:a04bd9e9688b69820cf22366defeaf9248700ad47e87133215a7e5811f0204c1 Committed
sha256:97124a0da9e9a24a5425acf487afe40dee074582b494be52d872fcfb1172df7e sha256:b914c4f7157b6e5f82530a9b4dfa537ba3b7c2903e2fdfa88b57c32504c8f3b3 Committed
sha256:987cb8ed593a5c025b8e9bbe2cc8b24f93b28e583f8f35ba64d5b25a6ea581d1                                                                         Committed
sha256:9a24f4c2114ddefbec763acebeb61ba30563db97ab554e13aa59cdcf12165707 sha256:52d58f02f883a2c3687215b71f7679b48d75f52564c248ade22d8ebfc0514b91 Committed
sha256:9aaf79c2f1043ddcbd161813aeb9794b08041d410003d418fca2c6e08ae70230 sha256:c7ba3bb588a182c39561fb18c21d85f2382b08ef29770265babe15852ef41471 Committed
sha256:9e4efa6e69767843fcf5bbeb87304dec035361c4ffb8484189c03f29a17bba1b                                                                         Committed
sha256:9f8bb936abbebb5cfb8240b8482d39bed88d0a4f0521a443e6f9b3051fdc0989 sha256:8ef2f4f683a86b6ee96207519e77296b52847f45f89572cde7a9edc7c1100ca5 Committed
sha256:a04bd9e9688b69820cf22366defeaf9248700ad47e87133215a7e5811f0204c1 sha256:269056aea1547ccb85a7e4b5aa0cfe362fbed722f14df4586fa1dcaba0dddb12 Committed
sha256:a10ec15ee5ef56b79fc519c6a56d77fc53a7d1add70e0809f0cd71f01ee4243e sha256:c7f4d36d93dda52de21b02f9b4b52b165c1a41375fbed14a8387868841181995 Committed
sha256:a1ea3fbe4d8a234d1a7f11c4717566efc9f9b8aa993cbe1f63242edbedf094ff sha256:cb60fb9b862c6a89f92e484bc3b72bbc0352b41166df5c4a68bfb52f52504a7d Committed
sha256:a639de4cb2d14e2e5450f16d6e10c45be109426ff0155b036712aa3d117f26cf sha256:87f5fe271fde4b62787033b985a5851a06e87a4a9d5a54f6bc6ee8b54db9e4d9 Committed
sha256:a726f89c78ec82f4307e818c5887d29d2bbc89cc8c07e92862e69d1aac5f3ddd sha256:be932f83f1da272e62d0e0e27ee2b39ce675d148364afe1846ffb9d8d8acd9cf Committed
sha256:acd6dec99d96a1d53173d713b6ee72f13edf346aa67eb357f8efb155697817f2 sha256:6e5daea76b043659b80e5d2b756c4d364fd663f683a70b7b2ec0aafb14056413 Committed
sha256:aed23d493e9662bea9f0b1ecb55f0438f8101a025cad07916277a83ba177571c sha256:79782f884ba606bd680107bcddc8e79e4e55d555942304c022dcf7bdadef23b7 Committed
sha256:af5105fb32284c3d381fec5ab2f66f0f09417f8519604dc89676ec16000d796f sha256:1e2d7a9c800c98052f5fa26d6c059d0762ace6d6e83624b9a3dc3e9063430506 Committed
sha256:b41b074d18a7d752bc046459e53d60f6b02f13872396f3e19b4482d341a4345f                                                                         Committed
sha256:b64283fc8b3b2640d7b72a94ae49e45e287befd9b59b76d62803851448a6c587 sha256:dab8c590d1236fb0f0aa5b15f0fe73c2515bb7e354a4f745c7e8426ee6f2f1a3 Committed
sha256:b71a12fa31024ac7e5e7f5a532a6665893c3482b470f7a98cf5625d438168cb1 sha256:84ab39a267f4d29b82795947a07b19ec459f8333a8869f8da4b6fbb15a2cc690 Committed
sha256:b8558647455474d3f08d3fbc59beda69b2c4dfb2d9080c8f9e8593f167b5f6a1                                                                         Committed
sha256:b914c4f7157b6e5f82530a9b4dfa537ba3b7c2903e2fdfa88b57c32504c8f3b3                                                                         Committed
sha256:bd56d64a4768ae109c1b382a438262fc2df1896a1a4858deaae285cd18f3d1c3 sha256:8aa2c77f6b0573ac718a7468d4d47cbf09963364d0c722906f9fac5e9c89186b Committed
sha256:bd85a340ea268d71b49c16a693f1764267228a5787fb0da2b81dc6c19fa9f97f sha256:89fafeeb9db96ca37f3214c0d14ce160ce82041b284ecf3a26641d3b3b5359e5 Committed
sha256:be8757e8c9a30bcf602c202290b903820fa890a108bf1c03f06e11db2e3cf9e6 sha256:52d58f02f883a2c3687215b71f7679b48d75f52564c248ade22d8ebfc0514b91 Committed
sha256:be8def87eb228e40250344b611afbd18a24264bc6aa1326c878732c7b91f6962 sha256:d1ce96c34f00e1937bedda321e521a141c15bb31d2b4c73c26d1c50a0a3dd4cc Committed
sha256:be932f83f1da272e62d0e0e27ee2b39ce675d148364afe1846ffb9d8d8acd9cf                                                                         Committed
sha256:bfb33b7e28ba1496e1f14c350aea3c8aa956fa5d96c43de3c886ba10c3750339 sha256:c124b2c4d0c2ffb1abf039e47ad041f55a876ed1e0060f58e6e0ac8e0588e782 Committed
sha256:c124b2c4d0c2ffb1abf039e47ad041f55a876ed1e0060f58e6e0ac8e0588e782                                                                         Committed
sha256:c2174a35d0a3a1324f91b3d29bcaf3270c966ae6d0f82c51f048c1511ee0561f sha256:e56b4a646c9bf0156587840461d73469bbf26055d7371447b4cb513819fb22ac Committed
sha256:c4a3048c8df8503d9c47b39263ba3cc855e7488bfa5cfd253e6d3e808abe6d88 sha256:cbe0942738d927779da92fb00fd3a89ee0749bd0c843b7899adb12709ebef707 Committed
sha256:c5ac479a4d171506ef5273b6c5f823102a3bb8710eec55613694e0dd7a91a5bb sha256:3f49d783f3f39276404439babcdba36ee338634c40a9a5a8a28df522d24d0c30 Committed
sha256:c7ba3bb588a182c39561fb18c21d85f2382b08ef29770265babe15852ef41471 sha256:724c3e4ae4a0d3094154899d85b59c3e776b4a6267103cd4ff715ff17bcd0afd Committed
sha256:c7f4d36d93dda52de21b02f9b4b52b165c1a41375fbed14a8387868841181995 sha256:7668a5eb0f0716d07ec2f3b0e12d431405311557ce41e61f881cfdc125ca931c Committed
sha256:ca52dc50a03a62727c4337675c2f3a66c8b0a82c98d53fe375eb7ca2a141a20c sha256:90df11b510893c99ddcd4549452390f8947b084cec1e301270926a4a8f77a1dd Committed
sha256:cb4596cc145400fb1f2aa56d41516b39a366ecdee7bf3f9191116444aacd8c90                                                                         Committed
sha256:cb60fb9b862c6a89f92e484bc3b72bbc0352b41166df5c4a68bfb52f52504a7d                                                                         Committed
sha256:cbe0942738d927779da92fb00fd3a89ee0749bd0c843b7899adb12709ebef707 sha256:af5105fb32284c3d381fec5ab2f66f0f09417f8519604dc89676ec16000d796f Committed
sha256:cc2447e1835a40530975ab80bb1f872fbab0f2a0faecf2ab16fbbb89b3589438                                                                         Committed
sha256:cc4fb2df65e7411567ec46817dd2a2d2d121b31680fafcf83a0e305c68cfd46d sha256:cb60fb9b862c6a89f92e484bc3b72bbc0352b41166df5c4a68bfb52f52504a7d Committed
sha256:ceab763bda4111def57f355c5009b948aea472c1e1a0ad1a6fd7df8be2b4ee97 sha256:6eb39a4c45053b99d4562e2bf1c6cf2f13b9cfb634447c6cdbf21b55c822ac93 Committed
sha256:cf1ac4276aa1377bd7dfb060bf8beea204a677cb86bb5aaf5f81c1e04aae1f8d sha256:b914c4f7157b6e5f82530a9b4dfa537ba3b7c2903e2fdfa88b57c32504c8f3b3 Committed
sha256:cf2374d14da73323d52ca7710656bf3dd9f2721243de439c3fefca2061b78af9 sha256:07d1df5250fa19cbba9e857aabcc302f323f33cb6eda5bbecf4645f1dc704347 Committed
sha256:cfc0039532c6d251c6eb90fe887f1629209948fd4b39ec1d218254f4da87f88e sha256:04f0b72a99243f128a871acdc2d015cbbd37a1ae9119e0e6a754c29ddd86f5f8 Committed
sha256:d0157aa0c95a4cae128dab97d699b2f303c8bea46914dc4a40722411f50bb40e                                                                         Committed
sha256:d058b2e299906a3d6814ee8759dc2dc533f1315ded793156909e3a0226b66ef2 sha256:90aff515acf292d70d9ac429d3cfa134341a8c20f9cf689fae1cbd4ae0916c2b Committed
sha256:d13700a9546f2f7afcbe0fe33bd519097dfd723cfc3f6f1e35880d067a52fce3 sha256:cb60fb9b862c6a89f92e484bc3b72bbc0352b41166df5c4a68bfb52f52504a7d Committed
sha256:d1ce96c34f00e1937bedda321e521a141c15bb31d2b4c73c26d1c50a0a3dd4cc sha256:c4a3048c8df8503d9c47b39263ba3cc855e7488bfa5cfd253e6d3e808abe6d88 Committed
sha256:d9d63d5691f24fa635706bd0803dc116d31f57870cb41471ec62b7840e487132 sha256:4cfdb156caefdad3a177a48d86a7a5062801869e628ce4c38bc7ddf01b0fa9d1 Committed
sha256:dab8c590d1236fb0f0aa5b15f0fe73c2515bb7e354a4f745c7e8426ee6f2f1a3 sha256:1ce51ff80e698669a6891a13d3fc33b855adf85e604817deb4136f44482904cb Committed
sha256:dae824f8a496996230fe3922cc0497a7065fcaee37c4a2f763cb82aef89fb34c sha256:66fc065f4bb48bf1b0a925e17f8bc4183a5fc45a7ef6d997f40c9c7fd6c65bde Committed
sha256:dbdd730772922b9fcfc5b88c95617c21ae0b3d9beb274e9db234dfd2930a1d1a sha256:ceab763bda4111def57f355c5009b948aea472c1e1a0ad1a6fd7df8be2b4ee97 Committed
sha256:de1b174a0e25174b60881457db7ba1282ed3705bb24a9f5c8b7766cf2521e491 sha256:1d0506f41c89884eca81d92f80479ba493be75903ca1bbc12d25e8883d8cd65a Committed
sha256:de369c4e2ef0226507f9689d776a96f2d5005ff5f322b0501e6da8614a94ef53 sha256:30c4d9fc6bf52615784c3dbcaa808085feb8ae94229022beab70bfab5cc56100 Committed
sha256:de39c9d61b393e5ef354cd247bb2b83cdfff594cb8147b6b029abfa0323c1a22 sha256:9359f802c19ffa094cc2e41b3e3c608577c0a6ea2e79fc61a172432c1f9be5c7 Committed
sha256:e14da2d720e0784a15e2e1b04bb3ed2e1aa537131001dc8886aa57a90136689f sha256:922ea7b2b282174db2272a0e50acb4a6946ed61a108a48f0adcca5463268d929 Committed
sha256:e2fc87ba0086e7db714848379d05c58f8ec7a801beed60a14d9370c4c148092c                                                                         Committed
sha256:e50ea7ba0d96cc84932c911015cbdf69de2f57850666466eaa1a42d532e1bc46 sha256:dae824f8a496996230fe3922cc0497a7065fcaee37c4a2f763cb82aef89fb34c Committed
sha256:e56b4a646c9bf0156587840461d73469bbf26055d7371447b4cb513819fb22ac sha256:8e3473c30af5cfb0c0a3cc6eb7cae4622a84098987e3ee02a420bcb11e84407b Committed
sha256:e5a8a3ef277f4e6738e6c89125857073f570831235bec75e580d289f6d9ad12a sha256:cf2374d14da73323d52ca7710656bf3dd9f2721243de439c3fefca2061b78af9 Committed
sha256:e916069494b485cb0ac31864cad59236e24ca497399e0a9d50ac5742ce012b28 sha256:8a8c5bd45d812d57aa49d9f8e61e8cd7dbad31f4eba95ab5246fca8985a38b50 Committed
sha256:e998b554f2d9e7d57713d4c78a5baf8da288990f487b37d0671c067c2a7f7b71 sha256:0e04e159318a93fa02c1615e565f4ad8ae6a28f7c3b306ebc78366f01d82b7f9 Committed
sha256:eaad9e85d5e63d602c26be64792c284cd222daac0ed3a21fc193ef59b883fc4f sha256:493f93de45de060d0dfc7c49f5a409a6b2f743b7fb0deadfb74a40b3a50ef4c1 Committed
sha256:eaea05884bde4441c0a8a223affa360a8372eedf75bbb4b14025e2a113858c62 sha256:3b891a13ddc94b26b977c6fe49461294d8a5a4ce731d8d273c24048a67f25c8a Committed
sha256:ebdee40a13a8a535d59ad25ba2f62e4d436c4d7081e80a44e5b884474f795580 sha256:2c36830d2b3b9931c89fbb58955e6a155bc13b084fbb6a0746c88e0f9e31dbac Committed
sha256:ec22d9ca60728c69acca7909d96922dfe5ceb4665ecd6c91b3282f18c3a58424 sha256:3745a2d825305d88cd6b4aed9b5b2de8eb2a36d237dd471368d18cf5d96e907f Committed
sha256:ede59ced70e61f7a85dc7e74d4a3bf7c6aea92274afc15fad005b23845b64430 sha256:a639de4cb2d14e2e5450f16d6e10c45be109426ff0155b036712aa3d117f26cf Committed
sha256:f08f0e2f3cd0106f219789426127ab0b99daa24ff236904c0c3b3d72fb8f25ee sha256:8ca1271bd3b844e4ba07eb1b52fa4d53ee1e1bf39b6d24eaeb88abe42119de0b Committed
sha256:f3e90bc2dc75fb4104e8e894771b2e35a22b2e3e9b0e716e7cfd2faf3687d3bb sha256:cb60fb9b862c6a89f92e484bc3b72bbc0352b41166df5c4a68bfb52f52504a7d Committed
sha256:f4d5b6c9593bbef07f183d0524f07fddff0120eb138042e16c46eb75d2355da2                                                                         Committed
sha256:f5ecd8d7cf73848fa7a5532706439062c198972cfbc757dbf4f3016fccab2172 sha256:3b891a13ddc94b26b977c6fe49461294d8a5a4ce731d8d273c24048a67f25c8a Committed
sha256:f79c4584d9c2c73bb9cc8d438a8d5c50ea7063df6baacb5d1423b6538a48e0ef sha256:26e009c8dab75b9f2b9e38c6887f031d5ee0b39c15885cee73324f7c638445c7 Committed
sha256:f7cdaa8664dcb7281f3d71f98a3bd29296a11ee542e12f380d509ec2b196f22e sha256:9f8bb936abbebb5cfb8240b8482d39bed88d0a4f0521a443e6f9b3051fdc0989 Committed
sha256:fd6089bf399876168a847c48fc04d7ca8a29bc0e158808cea991cb5ed79ba532 sha256:5a65d21aeee5bc0cdbb39b73f14304c111118c700125f3d97572498e9ee6a7de Committed
sha256:fef9a9ae4bfd07f361144f7acd9c2f768c9670a2eee895f9555f5bd5f960eb6e sha256:383530c0ccf430e191a4d441649090235a8a4b0a23f0e9e1af5afc29cbe89975 Committed
e-minguez commented 1 year ago

Not sure if related to https://github.com/containerd/containerd/issues/3369

After doing this, it seems to work:

# cd /var/lib/rancher/rke2/agent/containerd/
# ls
bin               io.containerd.grpc.v1.cri   io.containerd.runtime.v2.task            io.containerd.snapshotter.v1.native     lib
containerd.log            io.containerd.metadata.v1.bolt  io.containerd.snapshotter.v1.btrfs           io.containerd.snapshotter.v1.overlayfs  tmpmounts
io.containerd.content.v1.content  io.containerd.runtime.v1.linux  io.containerd.snapshotter.v1.fuse-overlayfs  io.containerd.snapshotter.v1.stargz
# rm -Rf *
# cd /run/containerd
# ls
runc  s
# rm -Rf *
# systemctl restart rke2-server
❯ k get po -o wide |grep -i alpine
alpine-hx6wq     0/1     CreateContainerError   0          9m18s   10.42.49.120      host1rke2   <none>           <none>
alpine-swjgp     1/1     Running                0          9m18s   10.42.97.244      host3rke2   <none>           <none>
alpine-xn4k2     1/1     Running                0          9m18s   10.42.204.87      host2rke2   <none>           <none>

❯ k get events | grep -i alpine-swjgp
9m49s       Normal    Scheduled                         pod/alpine-swjgp     Successfully assigned default/alpine-swjgp to host3rke2
9m48s       Normal    AddedInterface                    pod/alpine-swjgp     Add eth0 [10.42.97.232/32] from k8s-pod-network
8m6s        Normal    Pulling                           pod/alpine-swjgp     Pulling image "alpine"
9m48s       Normal    Pulled                            pod/alpine-swjgp     Successfully pulled image "alpine" in 566.285381ms (566.337168ms including waiting)
8m20s       Warning   Failed                            pod/alpine-swjgp     Error: failed to create containerd container: failed to stat parent: stat /var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/214/fs: no such file or directory
9m46s       Normal    Pulled                            pod/alpine-swjgp     Successfully pulled image "alpine" in 645.881241ms (645.902864ms including waiting)
9m31s       Normal    Pulled                            pod/alpine-swjgp     Successfully pulled image "alpine" in 583.042871ms (583.05558ms including waiting)
9m16s       Normal    Pulled                            pod/alpine-swjgp     Successfully pulled image "alpine" in 663.846146ms (663.858935ms including waiting)
9m1s        Normal    Pulled                            pod/alpine-swjgp     Successfully pulled image "alpine" in 585.318534ms (585.328178ms including waiting)
8m48s       Normal    Pulled                            pod/alpine-swjgp     Successfully pulled image "alpine" in 624.550162ms (624.578404ms including waiting)
8m34s       Normal    Pulled                            pod/alpine-swjgp     Successfully pulled image "alpine" in 579.501504ms (579.543858ms including waiting)
8m20s       Normal    Pulled                            pod/alpine-swjgp     Successfully pulled image "alpine" in 578.490662ms (578.506907ms including waiting)
2m12s       Normal    AddedInterface                    pod/alpine-swjgp     Add eth0 [10.42.97.244/32] from k8s-pod-network
2m12s       Normal    Pulling                           pod/alpine-swjgp     Pulling image "alpine"
2m4s        Normal    Pulled                            pod/alpine-swjgp     Successfully pulled image "alpine" in 7.717935937s (7.717945822s including waiting)
2m2s        Normal    Created                           pod/alpine-swjgp     Created container alpine
2m2s        Normal    Started                           pod/alpine-swjgp     Started container alpine
9m49s       Normal    SuccessfulCreate                  daemonset/alpine     Created pod: alpine-swjgp
e-minguez commented 1 year ago

On the other host if I just delete the container image with crictl, it works as well:

host1rke2:~ # crictl --config /var/lib/rancher/rke2/agent/etc/crictl.yaml images | grep -i alpine
docker.io/library/alpine                                                      latest                                     8ca4688f4f356       3.41MB
host1rke2:~ # crictl --config /var/lib/rancher/rke2/agent/etc/crictl.yaml rmi docker.io/library/alpine
Deleted: docker.io/library/alpine:latest
❯ k get po -o wide | grep alpine
alpine-hx6wq     1/1     Running       0          18m     10.42.49.120      host1rke2   <none>           <none>
alpine-swjgp     1/1     Running       0          18m     10.42.97.244      host3rke2   <none>           <none>
alpine-xn4k2     1/1     Running       0          18m     10.42.204.87      host2rke2   <none>           <none>

❯ k events | grep alpine-hx6wq
18m                      Normal    Scheduled                         Pod/alpine-hx6wq     Successfully assigned default/alpine-hx6wq to host1rke2
18m                      Normal    SuccessfulCreate                  DaemonSet/alpine     Created pod: alpine-hx6wq
18m                      Normal    AddedInterface                    Pod/alpine-hx6wq     Add eth0 [10.42.49.120/32] from k8s-pod-network
18m                      Normal    Pulled                            Pod/alpine-hx6wq     Successfully pulled image "alpine" in 832.941601ms (832.949763ms including waiting)
18m                      Normal    Pulled                            Pod/alpine-hx6wq     Successfully pulled image "alpine" in 596.542248ms (596.555577ms including waiting)
17m                      Normal    Pulled                            Pod/alpine-hx6wq     Successfully pulled image "alpine" in 570.429288ms (570.440455ms including waiting)
17m                      Normal    Pulled                            Pod/alpine-hx6wq     Successfully pulled image "alpine" in 635.101842ms (635.108913ms including waiting)
17m                      Normal    Pulled                            Pod/alpine-hx6wq     Successfully pulled image "alpine" in 639.277246ms (639.289284ms including waiting)
17m                      Normal    Pulled                            Pod/alpine-hx6wq     Successfully pulled image "alpine" in 566.555975ms (566.570346ms including waiting)
16m                      Normal    Pulled                            Pod/alpine-hx6wq     Successfully pulled image "alpine" in 564.755373ms (564.774742ms including waiting)
16m                      Normal    Pulled                            Pod/alpine-hx6wq     Successfully pulled image "alpine" in 573.263496ms (573.271779ms including waiting)
16m (x8 over 18m)        Warning   Failed                            Pod/alpine-hx6wq     Error: failed to create containerd container: failed to stat parent: stat /var/lib/rancher/rke2/agent/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs: no such file or directory
2m56s (x68 over 18m)     Normal    Pulling                           Pod/alpine-hx6wq     Pulling image "alpine"

Can this maybe related https://github.com/containerd/containerd/issues/3671 ?

brandond commented 1 year ago

if I just delete the container image with crictl, it works as well

I have seen also this fail sometimes. It really depends on how the snapshotter was corrupted, and whether or not the layer is shared with any other image. If it's shared, just deleting the one image won't clean it up, you'd need to identify and remove/repull all images that use that layer. Unless you have a surfeit of time on your hands, its usually easier to just nuke the directory and repull what's needed.

I'm going to close this, as any issues with this functionality should be tracked with containerd. RKE2 itself is not responsible for ensuring the integrity of the snapshotter filesystem.

e-minguez commented 12 months ago

https://github.com/containerd/containerd/pull/9401

Related fix