CC | Run the current tests offloading the image to the guest without the forked containerd

[fidencio]

SSIA.

I'm opening the PR, but the majority of the content was written by (and the credits are properly given to) @ChengyuZhu6.

PLEASE, DO NOT RUN /test HERE YET.

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[fidencio]

/fidencio-test

[stevenhorsman]

/fidencio-test

[stevenhorsman]

/fidencio-test

[stevenhorsman]

/fidencio-test

[stevenhorsman]

/fidencio-test

[stevenhorsman]

Chengyu - just to give you an update on what I've been trying today. The test that was 12 was still failing and after printing the containerd, kata and kubelet logs I couldn't spot an error, so I wondered if the problem was due to the fact that nydus snapshotter had already cached the docker config.json, so the missing credentials weren't a problem. I didn't know how to reset the credentials, so I tried moving the test above the others, but in http://jenkins.katacontainers.io/view/CCv0/job/tests-CCv0-ubuntu-20.04-x86_64-CC_CRI_CONTAINERD_K8S-IMAGE_OFFLOAD_TO_GUEST-PR/28/consoleFull it still fails and I can't see an obvious error related to the image pull failing on the host, so we might need to look elsewhere. I'll spend more time debugging tomorrow, but wanted to let you know what I've done in case you have any ideas whilst I'm asleep.

[ChengyuZhu6]

https://github.com/kata-containers/tests/assets/117246449/ba5875eb-c0cb-4f7f-966f-001d5ae10c86

[ChengyuZhu6]

@stevenhorsman I have tested in local machine and I can find the log "failed to resolve reference "quay.io/kata-containers/confidential-containers-auth:test": unexpected status from HEAD request".

[ChengyuZhu6]

Consider printing the log after assert_pod_fail "${pod_config}"for debugging. This is because the image pulling process is initiated by assert_pod_fail. Another potential reason could be that the kubelet log exceeds a length of 100,000, preventing the display of the necessary log.

[ChengyuZhu6]

[stevenhorsman]

/fidencio-test

[stevenhorsman]

@stevenhorsman I have tested in local machine and I can find the log "failed to resolve reference "quay.io/kata-containers/confidential-containers-auth:test": unexpected status from HEAD request".

Locally I'm seeing

Unknown desc = failed to pull and unpack image \"quay.io/kata-containers/confidential-containers-auth:test\": failed to resolve reference \"quay.io/kata-containers/confidential-containers-auth:test\": pulling from host quay.io failed with status code [manifests test]: 401 UNAUTHORIZED" image="quay.io/kata-containers/confidential-containers-auth:test"

but I've fixed the logging now, so we'll see what the automation comes up with and pick the best message :)

[stevenhorsman]

@stevenhorsman I have tested in local machine and I can find the log "failed to resolve reference "quay.io/kata-containers/confidential-containers-auth:test": unexpected status from HEAD request".

Locally I'm seeing

Unknown desc = failed to pull and unpack image \"quay.io/kata-containers/confidential-containers-auth:test\": failed to resolve reference \"quay.io/kata-containers/confidential-containers-auth:test\": pulling from host quay.io failed with status code [manifests test]: 401 UNAUTHORIZED" image="quay.io/kata-containers/confidential-containers-auth:test"

but I've fixed the logging now, so we'll see what the automation comes up with and pick the best message :)

I see the following messages in the CI logs as well:

10:41:10 # Sep 26 09:39:11 ubuntu20-f89750 kubelet[174155]: E0926 09:39:11.193013  174155 remote_image.go:236] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"quay.io/kata-containers/confidential-containers-auth:test\": failed to resolve reference \"quay.io/kata-containers/confidential-containers-auth:test\": pulling from host quay.io failed with status code [manifests test]: 401 UNAUTHORIZED" image="quay.io/kata-containers/confidential-containers-auth:test"

10:41:10 # Sep 26 09:40:45 ubuntu20-f89750 containerd[191826]: time="2023-09-26T09:40:45.346872399Z" level=error msg="PullImage \"quay.io/kata-containers/confidential-containers-auth:test\" failed" error="failed to pull and unpack image \"quay.io/kata-containers/confidential-containers-auth:test\": failed to resolve reference \"quay.io/kata-containers/confidential-containers-auth:test\": pulling from host quay.io failed with status code [manifests test]: 401 UNAUTHORIZED"

so I'll work on fixing up the grep to try and match that properly

[stevenhorsman]

/fidencio-test

[stevenhorsman]

/fidencio-test

[stevenhorsman]

/fidencio-test

[stevenhorsman]

@fidencio @stevenhorsman . Should we add encrypted image tests in pulling image in the guest with snapshotter? I believe the encrypted image tests would be passed with snapshotter to pull image in the guest.

So I think the plan beyond this draft PR is to switch the old jobs (that run the encrypted tests) to use image offload in https://github.com/confidential-containers/operator/compare/main...fidencio:cc-operator:remote_snapshotter?expand=1, so that should cover this. I think we are keeping this open for the pull on host testing

[ChengyuZhu6]

Oh sorry. I found the tests have included encrypted tests.

[ChengyuZhu6]

@fidencio @stevenhorsman The reason the tests for pulling images on the host are failing is because the latest version of nydus-image (2.2.3) does not include the tar-tarfs feature. Therefore, we need to build the main version of nydus-image from the nydus repository, similar to how we install nydus-snapshotter. It's important to note that this action will not impact the image offload tests.

23:45:33 # Sep 26 15:43:44 ubuntu20-384790 snapshotter[191643]: time="2023-09-26T15:43:44.034337972Z" level=warning msg="nydus image exec failed, error: 'tar-tarfs' isn't a valid value for '--type <type>'\n  [possible values: directory, dir-rafs, estargz-rafs, estargz-ref, estargztoc-ref, tar-rafs, targz-rafs, targz-ref, stargz_index]\n\n  Did you mean 'tar-rafs'?\n\nFor more information try '--help'\n"

[fidencio]

/fidencio-test