Dynamic changes in ConfigMap/Secrets are not propagated to volumes in a peer pod

[yoheiueda]

Kubernetes ConfigMap/Secrets support dynamic updates of their contents. When contents of a ConfigMap/Secret are updated, the updated contents must eventually propagated to a corresponding volume of the pod.

In the case of peer pods, however, the contents of ConfigMap/Secret are copied only once at pod creation time, and never copied again when they are updated.

This is because the current kata shim CCv0 implementation only copies the volume contents once. https://github.com/kata-containers/kata-containers/blob/b91337afef54d94e7f6458a0d1548e921dab6ea0/src/runtime/virtcontainers/fs_share_linux.go#L270

I heard that there is a mechanism to detect filesystem changes in kata shim for another purpose, so we may extend the volume copy using the detection mechanism. https://github.com/kata-containers/kata-containers/blob/69bf7e7519a06467dc8299347c689ac7663955c6/docs/design/inotify.md

So, I think this can be fixed in the kata shim side, and fixing this issue in cloud-api-adaptor side needs some tricks.

[yoheiueda]

@bpradipt pointed out that the Watchable Mount mechanism does not work for our case. The kata agent watches file changes, but files visible from kata agent on a peer pod VM will not change, since they are copied only once at startup.

We can work around this issue from kata shim or CAA.

If work around it on the CAA side, we can launch a goroutine per container just after the CAA proxy successfully calls CreateContainer. https://github.com/confidential-containers/cloud-api-adaptor/blob/9ec0295b657258a03a66fba5b9cab425d35f501a/pkg/adaptor/proxy/service.go#L163

The goroutine will monitor file changes under volume directories on the host side using fsnotify, and send file updates to kata-agent using copyFile.

volumes directories are located at /var/lib/kubelet/pods/ on the host side, and have the names like below.

• volumes/kubernetes.io~configmap
• volumes/kubernetes.io~secrets
• volumes/kubernetes.io~projected
• volume-subpaths
• etc-hosts

We can detect volume mounts by using pattern matching.

[bpradipt]

The changes could be made in the kata containers runtime or cloud-api-adaptor.

If implementing in CAA, the following info will be needed:

• pod ID to create the fsNotify watches - /var/lib/kubelet/pods/pod_id/volumes/
• Ability to map the file change event source to the destination file. The CopyFile grpc API doesn't include src path so we need to identify the right destination based on pattern matching. For example if the event source on the host is /var/lib/kubelet/pods/bc892ea0-de3c-4da8-a508-1ffcfaa723c1/volumes/kubernetes.io~downward-api/fooinfo/..2023_02_15_06_59_45.3837186726 then the destination will have /run/kata-containers/shared/containers/0e65a6d08a29b28ee43de2cd8a98187a151c90e6bed4f6e095202fefdad01870-3a867eacc041e1f3-fooinfo/..data The common pattern is the dir fooinfo. Not sure if updating the CopyFile api to include the full source path will help or not.
• Since CAA runs as a container, we'll need to mount the hostPath volume /var/lib/kubelet/pods

I have a sample working implementation on the kata containers side - https://github.com/bpradipt/kata-containers/tree/fsnotify

Will explore CAA implementation to have concrete details for comparison.

[yoheiueda]

I further investigated the feasibility of the CAA-side implementation, and it turned out that CAA does not have information of the source path information on the worker node, so I think it is difficult to implement the mechanism in CAA.

kata-shim translates a volume source path information for the worker node to one for the peer pod, and when CAA receives a CreateContainer request, it only contains volume source path information for the peer pod. CAA can't find out the original source path information for the worker node, so it can't detect the latest contents of the volume on the worker node.

Because of this reason, I think implementing the mechanism in kata-shim is a better approach.

[huoqifeng]

Agree, change in kata-shim will also reduce the conflict risk between Kata/CoCo and CAA.

[bpradipt]

While the core enablement PR is in kata-containers, an e2e test for this scenario needs to be added. The e2e test should be added to the following code - https://github.com/confidential-containers/cloud-api-adaptor/blob/staging/test/e2e/common_suite.go

cc @stevenhorsman @littlejawa

[mkulke]

Are there implications for the TEE in a peer-pod? If we collect evidence about the environment and appraise those via policy only initially at startup and then change e.g. volume content later? Do we have to re-attest?

[stevenhorsman]

I think @magowan might have had some thoughts about this with confidential containers as well?