Cannot deploy the operator through minikube: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock

[gxcuit]

Describe the bug Hi, recently I was trying the coco quick start guide. I noticed that it said we can use Minikube cluster to try the CoCo.

I start the minikube with containerd runtime and docker vm-driver through the following command:

minikube start --vm-driver=docker --container-runtime=containerd --image-mirror-country='cn' --force

Everything looks fine before I deploy the Nginx example: I successfully deployed the operator, the custom resources(https://github.com/confidential-containers/documentation/blob/v0.1.0/quickstart.md#create-the-custom-resource)

And I got the corresponding runtime class:

NAME            HANDLER         AGE
kata            kata            9m55s
kata-clh        kata-clh        9m55s
kata-clh-tdx    kata-clh-tdx    9m55s
kata-qemu       kata-qemu       9m55s
kata-qemu-tdx   kata-qemu-tdx   9m55s
kata-qemu-sev   kata-qemu-sev   9m55s

However, when I deployed the bitnami/nginx image, I got the following error:

Type     Reason                  Age                     From     Message
  ----     ------                  ----                    ----     -------
  Warning  FailedCreatePodSandBox  2m29s (x2641 over 32h)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: Failed to Check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 2711030719:1024: unknown

Does anyone know the reason?

System info:

Frdora with 5.17.5-300.fc36.x86_64 (Also tried Ubuntu with the same error)

minikube version: v1.28.0

[root@fedora ~]# minikube profile list
|----------|-----------|------------|--------------|------|---------|---------|-------|--------|
| Profile  | VM Driver |  Runtime   |      IP      | Port | Version | Status  | Nodes | Active |
|----------|-----------|------------|--------------|------|---------|---------|-------|--------|
| minikube | docker    | containerd | 192.168.49.2 | 8443 | v1.25.3 | Running |     1 | *      |
|----------|-----------|------------|--------------|------|---------|---------|-------|--------|

[fidencio]

While I'm testing this right now, I'd like to check whether you can see any difference on using a different driver, such as kvm2 for the minikube.

[fidencio]

I was finally able to get minikube up and running with the docker driver and I did, indeed, hit the issue with the kata-qemu runtime class.

Events:
  Type     Reason                  Age   From               Message
  ----     ------                  ----  ----               -------
  Normal   Scheduled               42s   default-scheduler  Successfully assigned default/nginx-kata-qemu to minikube
  Warning  FailedCreatePodSandBox  9s    kubelet            Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: Failed to Check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 3189232285:1024: unknown

It's worth mentioning that using the kata-clh one things work as expected.

Let me debug what's happening with QEMU here.

[bpradipt]

Describe the bug Hi, recently I was trying the coco quick start guide. I noticed that it said we can use Minikube cluster to try the CoCo.

I start the minikube with containerd runtime and docker vm-driver through the following command:

minikube start --vm-driver=docker --container-runtime=containerd --image-mirror-country='cn' --force

Everything looks fine before I deploy the Nginx example: I successfully deployed the operator, the custom resources(https://github.com/confidential-containers/documentation/blob/v0.1.0/quickstart.md#create-the-custom-resource)

And I got the corresponding runtime class:

NAME            HANDLER         AGE
kata            kata            9m55s
kata-clh        kata-clh        9m55s
kata-clh-tdx    kata-clh-tdx    9m55s
kata-qemu       kata-qemu       9m55s
kata-qemu-tdx   kata-qemu-tdx   9m55s
kata-qemu-sev   kata-qemu-sev   9m55s

However, when I deployed the bitnami/nginx image, I got the following error:

Type     Reason                  Age                     From     Message
  ----     ------                  ----                    ----     -------
  Warning  FailedCreatePodSandBox  2m29s (x2641 over 32h)  kubelet  (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to create containerd task: failed to create shim task: Failed to Check if grpc server is working: rpc error: code = DeadlineExceeded desc = timed out connecting to vsock 2711030719:1024: unknown

Does anyone know the reason?

System info:

Frdora with 5.17.5-300.fc36.x86_64 (Also tried Ubuntu with the same error)

minikube version: v1.28.0

[root@fedora ~]# minikube profile list
|----------|-----------|------------|--------------|------|---------|---------|-------|--------|
| Profile  | VM Driver |  Runtime   |      IP      | Port | Version | Status  | Nodes | Active |
|----------|-----------|------------|--------------|------|---------|---------|-------|--------|
| minikube | docker    | containerd | 192.168.49.2 | 8443 | v1.25.3 | Running |     1 | *      |
|----------|-----------|------------|--------------|------|---------|---------|-------|--------|

Which runtimeClass are you using ? Note that Qemu doesn't work with Kind or Minikube. As @fidencio mentioned clh works.

[fidencio]

I'd add this as a known issue that we should dig deeper in the future, but not a blocker for the v0.2.0. We should document somewhere that minikube will not work with QEMU runtime classes.

[bpradipt]

@gxcuit can you please confirm if your issue was specific to using qemu runtime class with minikube ?

[gxcuit]

@gxcuit can you please confirm if your issue was specific to using qemu runtime class with minikube ?

@bpradipt Hi This issue is specific to using kata runtime class with minikube. If I switch to kata-clh as suggested by @fidencio, it works.

minikube start --driver=docker --container-runtime=containerd

Do I need to install the ccv0-containerd? I install the original docker on my computer.

[bpradipt]

@gxcuit unfortunately kata runtime class (which defaults to using qemu) doesn't work with minikube and kind. Basically Kata qemu VM creation inside the container running Kubernetes fails and this needs to be investigated.

I have created an issue in https://github.com/kata-containers/kata-containers/issues/5692 We can use the above issue to continue debugging and taking help from qemu experts.

@gxcuit @fidencio wdyt ?

[gxcuit]

wdyt

@bpradipt When I follow this guide installing kata with minikube, everything works fine(with kvm2 driver). But when I use that same command(minikube start --vm-driver kvm2 --container-runtime containerd) to install ccv0-kata, it failed. Later I will try again and upload the logs.