search

kata-containers / tests

Kata Containers tests, CI, and metrics
https://katacontainers.io/
Apache License 2.0
140 stars 196 forks source link

`[cc][agent][kubernetes][containerd] Test can pull an encrypted image inside the guest with decryption key` is failing in the TDX CI #5276

Open fidencio opened 1 year ago

fidencio commented 1 year ago 
21:50:46 not ok 1 [cc][agent][kubernetes][containerd] Test can pull an encrypted image inside the guest with decryption key
21:50:46 # (from function `kubernetes_create_ssh_demo_pod' in file confidential/lib.sh, line 83,
21:50:46 #  in test file confidential/agent_image_encrypted.bats, line 40)
21:50:46 #   `kubernetes_create_ssh_demo_pod' failed
21:50:46 # No resources found in default namespace.
21:50:46 # Prepare containerd for Confidential Container
21:50:46 # time="2022-11-25T04:48:29+08:00" level=fatal msg="unable to determine runtime API version: rpc error: code = Unknown desc = server is not initialized yet"
21:50:46 # time="2022-11-25T04:48:34+08:00" level=fatal msg="unable to determine runtime API version: rpc error: code = Unknown desc = server is not initialized yet"
21:50:46 # Reconfigure Kata Containers
21:50:46 # Enable agent https proxy
21:50:46 #   % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
21:50:46 #                                  Dload  Upload   Total   Spent    Left  Speed
21:50:46 # 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100    63  100    63    0     0     68      0 --:--:-- --:--:-- --:--:--    68
21:50:46 # Cloning into '/var/lib/jenkins/workspace/kata-containers-CCv0-cloud-hypervisor-TDX-CRI_CONTAINERD_K8S-PR/go/src/github.com/confidential-containers/documentation'...
21:50:46 # service/ccv0-ssh created
21:50:46 # deployment.apps/ccv0-ssh created
21:50:46 # error: timed out waiting for the condition on pods/ccv0-ssh-9fd8559d8-w8p4f

And the relevant piece of log:

21:50:47 # Nov 25 04:50:35 tdx-ci kata[671135]: time="2022-11-25T04:50:35.449455915+08:00" level=debug msg="PullImage() start" image="docker.io/katadocker/ccv0-ssh:latest" name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=containerd-kata-shim-v2
21:50:47 # Nov 25 04:50:35 tdx-ci kata[671135]: time="2022-11-25T04:50:35.449523664+08:00" level=debug msg="Making image pull request" image="docker.io/katadocker/ccv0-ssh:latest" name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=containerd-kata-shim-v2
21:50:47 # Nov 25 04:50:35 tdx-ci kata[671135]: time="2022-11-25T04:50:35.455281139+08:00" level=debug msg="reading guest console" console-protocol=pty console-url=/dev/pts/0 name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=virtcontainers subsystem=sandbox vmconsole="{\"msg\":\"Attestation Agent already running\",\"level\":\"INFO\",\"ts\":\"2022-11-24T20:50:34.736159481Z\",\"source\":\"agent\",\"subsystem\":\"rpc\",\"version\":\"0.1.0\",\"pid\":\"131\",\"name\":\"kata-agent\"}"
21:50:47 # Nov 25 04:50:35 tdx-ci kata[671135]: time="2022-11-25T04:50:35.455347714+08:00" level=debug msg="reading guest console" console-protocol=pty console-url=/dev/pts/0 name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=virtcontainers subsystem=sandbox vmconsole=
21:50:47 # Nov 25 04:50:35 tdx-ci kata[671135]: time="2022-11-25T04:50:35.455368969+08:00" level=debug msg="reading guest console" console-protocol=pty console-url=/dev/pts/0 name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=virtcontainers subsystem=sandbox vmconsole="{\"msg\":\"enable_signature_verification set to: false\",\"level\":\"INFO\",\"ts\":\"2022-11-24T20:50:34.736202638Z\",\"version\":\"0.1.0\",\"source\":\"agent\",\"name\":\"kata-agent\",\"subsystem\":\"rpc\",\"pid\":\"131\"}"
21:50:47 # Nov 25 04:50:35 tdx-ci kata[671135]: time="2022-11-25T04:50:35.455400741+08:00" level=debug msg="reading guest console" console-protocol=pty console-url=/dev/pts/0 name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=virtcontainers subsystem=sandbox vmconsole=
21:50:47 # Nov 25 04:50:35 tdx-ci kata[671135]: time="2022-11-25T04:50:35.455418606+08:00" level=debug msg="reading guest console" console-protocol=pty console-url=/dev/pts/0 name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=virtcontainers subsystem=sandbox vmconsole="{\"msg\":\"pull image \\\"ccv0-ssh_latest\\\", bundle path \\\"/run/kata-containers/ccv0-ssh_latest\\\"\",\"level\":\"INFO\",\"ts\":\"2022-11-24T20:50:34.736239697Z\",\"pid\":\"131\",\"subsystem\":\"rpc\",\"source\":\"agent\",\"version\":\"0.1.0\",\"name\":\"kata-agent\"}"
21:50:47 # Nov 25 04:50:35 tdx-ci kata[671135]: time="2022-11-25T04:50:35.455444843+08:00" level=debug msg="reading guest console" console-protocol=pty console-url=/dev/pts/0 name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=virtcontainers subsystem=sandbox vmconsole=
21:50:47 # Nov 25 04:50:37 tdx-ci kata[671135]: time="2022-11-25T04:50:37.683643407+08:00" level=error msg="agent pull image err. rpc error: code = Internal desc = failed to pull manifest Not authorized: url https://registry-1.docker.io/v2/katadocker/ccv0-ssh/manifests/latest" name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=virtcontainers subsystem=kata_agent
21:50:47 # Nov 25 04:50:37 tdx-ci kata[671135]: time="2022-11-25T04:50:37.683693993+08:00" level=error msg="kata runtime PullImage err. rpc error: code = Internal desc = failed to pull manifest Not authorized: url https://registry-1.docker.io/v2/katadocker/ccv0-ssh/manifests/latest" name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=containerd-kata-shim-v2
21:50:47 # Nov 25 04:50:37 tdx-ci kata[671135]: time="2022-11-25T04:50:37.683711278+08:00" level=debug msg="PullImage() end" image="docker.io/katadocker/ccv0-ssh:latest" name=containerd-shim-v2 pid=671135 sandbox=f933ff261e8aa153de6c43e74e3ffbfb4796f5ee9c287582b9e57b1ae71cd136 source=containerd-kata-shim-v2
fidencio commented 1 year ago

/cc @arronwy

fidencio commented 1 year ago

I could see the failure happening with both Cloud Hypervisor and QEMU, and this didn't happen in the Merge PR on the tests side.

It may be a flake, but we need to debug it further.