search

awslabs / soci-snapshotter

A containerd snapshotter plugin which enables standard OCI images to be lazily loaded without requiring a build-time conversion step.
Apache License 2.0
534 stars 55 forks source link

failed to get reader from content store #1389

Closed vsoch closed 1 month ago

vsoch commented 1 month ago

Hiya! I am installing this via a daemonset (I can share the code if needed) but I am hitting this error:

Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               9s    default-scheduler  Successfully assigned default/soci-sample-deployment-584bf9b4f8-b2q4k to ip-192-168-21-204.us-east-2.compute.internal
  Warning  FailedCreatePodSandBox  9s    kubelet            Failed to create pod sandbox: rpc error: code = NotFound desc = failed to create containerd container: error unpacking image: failed to extract layer sha256:59b1469b8fbd05fd256959ad9d7d776b9937b848d75113a0d7c1af442528b6d0: failed to get reader from content store: content digest sha256:0692f38991d53a0c28679148f99de26a44d630fda984b41f63c5e19f839d15a6: not found

Here are the commands and output - they are run with nsenter -t 0 -m bash

+ SOCI_VERSION=0.7.0
+ INSTALL_FILE=/soci-install/install-soci.sh
+ [[ ! -f /soci-install/install-soci.sh ]]
+ echo 'Copying install files onto the host node'
+ cp /soci-install/install-soci.sh /mnt/install/install.sh
Copying install files onto the host node
+ chmod +x /mnt/install/install.sh
+ cp /soci-install/write_config.py /mnt/install/write_config.py
+ cp /soci-install/config.toml /mnt/install/config.toml
Executing nsenter to connect from container to host
+ echo 'Executing nsenter to connect from container to host'
+ nsenter -t 1 -m bash /mnt/install/install.sh 0.7.0
+ VERSION=0.7.0
+ [[ -f /etc/soci-snapshotter-grpc/install-complete.txt ]]
+ yum install fuse -y
Loaded plugins: priorities, update-motd, versionlock
Resolving Dependencies
--> Running transaction check
---> Package fuse.x86_64 0:2.9.2-11.amzn2 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package      Arch           Version                   Repository          Size
================================================================================
Installing:
 fuse         x86_64         2.9.2-11.amzn2            amzn2-core          86 k

Transaction Summary
================================================================================
Install  1 Package

Total download size: 86 k
Installed size: 218 k
Downloading packages:
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : fuse-2.9.2-11.amzn2.x86_64                                   1/1 
  Verifying  : fuse-2.9.2-11.amzn2.x86_64                                   1/1 

Installed:
  fuse.x86_64 0:2.9.2-11.amzn2                                                  

Complete!
+ python3 -m pip install toml
WARNING: Running pip install with root privileges is generally not a good idea. Try `python3 -m pip install --user` instead.
Collecting toml
  Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
Installing collected packages: toml
Successfully installed toml-0.10.2

...
 27700K .......... .......... .......... .......... .......... 99%  317M 0s
 27750K .......... .......... .......... .......... ......    100%  237M=0.1s

2024-10-05 04:24:35 (263 MB/s) - 'soci-snapshotter-0.7.0-linux-amd64.tar.gz' saved [28463452/28463452]

+ tar -xzvf soci-snapshotter-0.7.0-linux-amd64.tar.gz
NOTICE.md
THIRD_PARTY_LICENSES
soci
soci-snapshotter-grpc
+ chmod +x ./soci ./soci-snapshotter-grpc
+ mv ./soci /usr/local/bin/soci
+ mv /soci-snapshotter-grpc /usr/local/bin/soci-snapshotter-grpc
+ /usr/local/bin/soci-snapshotter-grpc --version
soci-snapshotter-grpc version v0.7.0 7c6fae2c3848fe8ad161ce35d3423898cea5fde8
+ wget https://raw.githubusercontent.com/awslabs/soci-snapshotter/refs/heads/main/soci-snapshotter.service
--2024-10-05 04:24:35--  https://raw.githubusercontent.com/awslabs/soci-snapshotter/refs/heads/main/soci-snapshotter.service
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.111.133, 185.199.108.133, 185.199.109.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.111.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1466 (1.4K) [text/plain]
Saving to: 'soci-snapshotter.service'

     0K .                                                     100% 34.7M=0s

2024-10-05 04:24:35 (34.7 MB/s) - 'soci-snapshotter.service' saved [1466/1466]

+ mv soci-snapshotter.service /usr/lib/systemd/system/soci-snapshotter.service
+ sudo systemctl daemon-reload
+ sudo systemctl enable --now soci-snapshotter
Created symlink from /etc/systemd/system/multi-user.target.wants/soci-snapshotter.service to /usr/lib/systemd/system/soci-snapshotter.service.
+ sudo systemctl status soci-snapshotter
● soci-snapshotter.service - soci snapshotter containerd plugin
   Loaded: loaded (/usr/lib/systemd/system/soci-snapshotter.service; enabled; vendor preset: disabled)
   Active: active (running) since Sat 2024-10-05 04:24:36 UTC; 17ms ago
     Docs: https://github.com/awslabs/soci-snapshotter
 Main PID: 7227 (soci-snapshotte)
   CGroup: /system.slice/soci-snapshotter.service
           └─7227 /usr/local/bin/soci-snapshotter-grpc

Oct 05 04:24:36 ip-192-168-21-204.us-east-2.compute.internal systemd[1]: Starting soci snapshotter containerd plugin...
Oct 05 04:24:36 ip-192-168-21-204.us-east-2.compute.internal soci-snapshotter-grpc[7227]: {"level":"info","msg":"starting soci-snapshotter-grpc","revision":"7c6fae2c3848fe8ad161ce35d3423898cea5fde8","time":"2024-10-05T04:24:36.148769749Z","version":"v0.7.0"}
Oct 05 04:24:36 ip-192-168-21-204.us-east-2.compute.internal soci-snapshotter-grpc[7227]: {"emitMetricPeriod":10000000000,"fetchPeriod":500000000,"level":"info","maxQueueSize":100,"msg":"constructing background fetcher","silencePeriod":30000000000,"time":"2024-10-05T04:24:36.502343141Z"}
Oct 05 04:24:36 ip-192-168-21-204.us-east-2.compute.internal soci-snapshotter-grpc[7227]: {"address":"/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock","level":"info","msg":"soci-snapshotter-grpc successfully started","time":"2024-10-05T04:24:36.504470699Z"}
Oct 05 04:24:36 ip-192-168-21-204.us-east-2.compute.internal systemd[1]: Started soci snapshotter containerd plugin.
+ python3 /mnt/install/write_config.py
version = 2
root = "/var/lib/containerd"
state = "/run/containerd"
imports = [ "/etc/containerd/config.d/*.toml",]

[grpc]
address = "/run/containerd/containerd.sock"

[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "602401143452.dkr.ecr.us-east-2.amazonaws.com/eks/pause:3.5"

[proxy_plugins.soci]
type = "snapshot"
address = "/run/soci-snapshotter-grpc/soci-snapshotter-grpc.sock"

[plugins."io.containerd.grpc.v1.cri".containerd]
default_runtime_name = "runc"
discard_unpacked_layers = true
snapshotter = "soci"
disable_snapshot_annotations = false

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"

[plugins."io.containerd.grpc.v1.cri".cni]
bin_dir = "/opt/cni/bin"
conf_dir = "/etc/cni/net.d"

[proxy_plugins.soci.exports]
root = "/var/lib/soci-snapshotter-grpc"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true

+ tail /etc/containerd/config.toml
conf_dir = "/etc/cni/net.d"

[proxy_plugins.soci.exports]
root = "/var/lib/soci-snapshotter-grpc"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"

[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
+ echo 'Restarting containerd - this will end the daemonset'
+ mkdir -p /etc/soci-snapshotter-grpc
Restarting containerd - this will end the daemonset
+ cp /mnt/install/config.toml /etc/soci-snapshotter-grpc/config.toml
+ touch /etc/soci-snapshotter-grpc/install-complete.txt
+ systemctl restart containerd
+ sleep infinity

Let me know what information you need to debug and I'm happy to help - likely this is a configuration setting that I need to tweak. I'm hoping to share this with you (and the larger community) when I finish. Thanks for the extra set of eyes!

vsoch commented 1 month ago

I'm wondering if it's related to https://github.com/awslabs/soci-snapshotter/blob/main/docs/kubernetes.md#limitations

vsoch commented 1 month ago

I got it working! But I need to figure out if I can automate it, will report back.

Kern-- commented 1 month ago

I double checked. The layer that that error is complaining about is the layer from the pause image, so yes, it looks related to the limitations where the pause image is being pulled with the wrong snapshotter. I'm guessing you came to the same conclusion since you managed to work around it?

Is there anything else needed here?

vsoch commented 1 month ago

Oh no, I figured this all out, sorry didn't follow up! Here is the post: https://vsoch.github.io/2024/container-pulling/ and my solution: https://github.com/converged-computing/soci-installer. Thanks!