search

openshift / sandboxed-containers-operator

An operator to enhance an Openshift/Kubernetes cluster to support running sandboxed containers
Apache License 2.0
37 stars 41 forks source link

peerpod: Remote hypervisor config files needs to be updated to align with upstream changes #426

Closed liudalibj closed 3 months ago

liudalibj commented 3 months ago

- Description of the problem which is fixed/What is the use case

CAA latest main branch codes are base on kata-agent main branch code already, it will always pull image in guest. Without this change the CAA latest main branch codes can't work well in OCP cluster, the error logs when create a peerpod look like:

Pulling image separately not support on main. It is required to use the nydus-snapshotter, which isn't configured properly here.

related code https://github.com/confidential-containers/cloud-api-adaptor/blob/main/src/cloud-api-adaptor/pkg/adaptor/proxy/service.go#L85C18-L85C146

- What I did

so we need enable the pull-in-guest function in crio config file 50-kata-remote:

[crio.runtime.runtimes.kata-remote]
 runtime_path = "/usr/bin/containerd-shim-kata-v2-tp"
 runtime_type = "vm"
 runtime_root = "/run/vc"
 runtime_config_path = "/opt/kata/configuration-remote.toml"
 privileged_without_host_devices = true

 runtime_pull_image = true
 allowed_annotations = [
 "io.kubernetes.cri-o.Devices",
]

- How to verify it

[root@bastion-ocp-cn-new ~]# oc get nodes NAME STATUS ROLES AGE VERSION control-plane-ocp-cn-new-0.ocp-cn-new.test-ocp-cn-new.coco Ready control-plane,master 6d20h v1.29.5+29c95f3 control-plane-ocp-cn-new-1.ocp-cn-new.test-ocp-cn-new.coco Ready control-plane,master 6d20h v1.29.5+29c95f3 control-plane-ocp-cn-new-2.ocp-cn-new.test-ocp-cn-new.coco Ready control-plane,master 6d20h v1.29.5+29c95f3 worker-ocp-cn-new-0.ocp-cn-new.test-ocp-cn-new.coco Ready kata-oc,worker 6d20h v1.29.5+29c95f3 worker-ocp-cn-new-1.ocp-cn-new.test-ocp-cn-new.coco Ready kata-oc,worker 6d20h v1.29.5+29c95f3 [root@bastion-ocp-cn-new ~]# 

- Install latest sandboxed-containers-operator

- login to worker nodes
    - update the file `/etc/crio/crio.conf.d/50-kata-remote` add one line ` runtime_pull_image = true`
    - restart crio
- Update CAA image to latest opensource tag base on the main branch
https://quay.io/repository/confidential-containers/cloud-api-adaptor?tab=tags
eg.

quay.io/confidential-containers/cloud-api-adaptor:dev-0480804f8f7d06202a5a18fef3a966ec6e4e59c7

- Build podvm image, make sure the embedded pause image is: "quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f3cb6f9d265291b47a7491c2ba4f4dd0752a18b661eee40584f9a5dbcbe13bb"
    run follow commands before build podvm image
rm -rf  /root/cloud-api-adaptor/src/cloud-api-adaptor/podvm-mkosi/resources/binaries-tree/pause_bundle
mkdir /tmp/pause
skopeo copy "docker://quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7f3cb6f9d265291b47a7491c2ba4f4dd0752a18b661eee40584f9a5dbcbe13bb" "oci:/tmp/pause:7f3cb6f9d265291b47a7491c2ba4f4dd0752a18b661eee40584f9a5dbcbe13bb" --authfile /root/auth.json

umoci unpack --rootless --image "/tmp/pause:7f3cb6f9d265291b47a7491c2ba4f4dd0752a18b661eee40584f9a5dbcbe13bb" /root/cloud-api-adaptor/src/cloud-api-adaptor/podvm-mkosi/resources/binaries-tree/pause_bundle
```

- Description for the changelog

add " runtime_pull_image = true" to "50-kata-remote" file to use the pull-in-guest function from crio.

openshift-ci[bot] commented 3 months ago

Hi @liudalibj. Thanks for your PR.

I'm waiting for a openshift member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.
bpradipt commented 3 months ago

@liudalibj thanks for this.. Can you also please update the config to add this stanza and update the commit description accordingly. Since the process requires transforming the plaintext via butane, I don't see any benefit of separate commits. But please add the description of the changes to the commit message.

 allowed_annotations = [
 "io.kubernetes.cri-o.Devices",
]

Complete config

[crio.runtime.runtimes.kata-remote]
 runtime_path = "/usr/bin/containerd-shim-kata-v2-tp"
 runtime_type = "vm"
 runtime_root = "/run/vc"
 runtime_config_path = "/opt/kata/configuration-remote.toml"
 privileged_without_host_devices = true

 runtime_pull_image = true
 allowed_annotations = [
 "io.kubernetes.cri-o.Devices",
]

Please also add the following to the commit message Fixes: #KATA-3155

bpradipt commented 3 months ago

/ok-to-test

openshift-ci[bot] commented 3 months ago

@liudalibj: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/sandboxed-containers-operator-e2e 982220541019493266d313764eb949e452db9a5b link false /test sandboxed-containers-operator-e2e
ci/prow/check 982220541019493266d313764eb949e452db9a5b link false /test check

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).