Failed to create pod sandbox: rpc error: code = NotFound desc = failed to create containerd container: create snapshot: missing parent

stevenhorsman commented 10 months ago

I was doing some testing with the operator and peer-pods and hit the error:

Events:
  Type     Reason                  Age                  From               Message
  ----     ------                  ----                 ----               -------
  Normal   Scheduled               3m28s                default-scheduler  Successfully assigned default/busybox-cc to 10.250.128.4
  Warning  FailedCreatePodSandBox  7s (x16 over 3m28s)  kubelet            Failed to create pod sandbox: rpc error: code = NotFound desc = failed to create containerd container: create snapshot: missing parent "k8s.io/2/sha256:500f1716fa0168cec7485186127dd1d869f216bf88054e3f0fa160c0b5144f36" bucket: not found

which was also seen in the SEV daily baseline: http://jenkins.katacontainers.io/view/Daily%20CCv0%20baseline/job/confidential-containers-operator-main-ubuntu-20.04_sev-x86_64-containerd_kata-qemu-sev-baseline/217/console

I was debugging another issues, so I don't have clean steps, but the rough run of events was:

Install the operator in my cluster (the peer pods variant)
Try to test pull on guest with the nydus snapshotter, which wasn't working, so I did a whole bunch of things related to that
Uninstall the operator
Install the operatpr
Create a basic busybox test pod This is where the error showed up - before the create container request for pause image, or busybox hit got to the cloud-api-adaptor

fitzthum commented 10 months ago

I have run into something similar. Try deleting (with crictl) the pause image along with any other images that you pull in your tests before installing the operator.

fitzthum commented 10 months ago

Maybe @chengyuzhu6 can elaborate. He pointed me towards the solution. I am wondering if this is something that we should document or try to fix.

stevenhorsman commented 10 months ago

I have run into something similar. Try deleting (with crictl) the pause image along with any other images that you pull in your tests before installing the operator.

This was on managed Kubernetes, so a bit tricky to do that (and we accidentally killed containerd in the process). I will try and reproduce this though.

fidencio commented 10 months ago

JFYI, I'm also hitting this with the TDX CI.

confidential-containers / operator

Failed to create pod sandbox: rpc error: code = NotFound desc = failed to create containerd container: create snapshot: missing parent #270