Replace `containerdConfigPatches` with an API that allows config updates to multiple files

[nicks]

What would you like to be added:

Replace containerdConfigPatches with an API that allows updates to multiple files

At this point, I'm not proposing a specific API. Mainly asking whether the KIND project would be open to a proposal, and if there's been prior work on what a proposal might look like.

Why is this needed:

The Kind config API has a field called containerdConfigPatches https://github.com/kubernetes-sigs/kind/blob/ac28d7fb19b4f353369d889b3900a7a9dd46f4c1/pkg/internal/apis/config/types.go#L66

This API made sense when the containerd config was a single TOML file.

Now, the containerd registry config is split across multiple files. For more info, see: https://github.com/containerd/containerd/blob/main/docs/cri/config.md#registry-configuration

We're currently working around this by patching the file post-hoc, like we do with docker exec commands in this PR: https://github.com/kubernetes-sigs/kind/pull/3245/files

[Romain-Geissler-1A]

Hi,

Some month later, was there any internal discussion between kind maintainers for a long term solution for this ? I am working in a corporate environment where most registries have to go through an internal mirror to be useable, and right now the deprecated v1 way to patch containerd seems easier to implement than the v2 where multiple files would need to be created/patched.

cc @BenTheElder

[aojea]

I can only see one file , am I missing something?

 docker exec -it kind-worker ls -al /etc/containerd
total 20
drwxr-xr-x 2 root root 4096 Feb 12 22:49 .
drwxr-xr-x 1 root root 4096 Mar 22 09:18 ..
-rw-r--r-- 1 root root 1738 Feb 12 22:44 config.toml
-rw-r--r-- 1 root root 3648 Feb 12 22:49 cri-base.json

[BenTheElder]

https://github.com/kubernetes-sigs/kind/issues/2875 has been the tracking issue for this.

I don't think we want to try to add a generalized file manipulation API, that will have a lot of pitfalls like intermediate directory creation, file modes / permissions / xattrs / ...

We have extraMounts currently instead, the reason for the exec/file approach is to attempt to bridge the transition between old and new devel scripts and this unfortunate containerd change that requires old or new fields but not both.

[Romain-Geissler-1A]

I am not sure how well is supported kind on a remote OCI engine, but extraMounts would assume the registry files are installed on the remote host where the OCI engine runs while so far the config was provided by the client. Maybe we are strange, but in our case the users have no real "control" on the remote OCI installation (though docker in rootful mode in theory is more or less equal to root access on the host). And the other alternative of re-creating images ourselves to add the needed file doesn't seem to be something that would easily be maintanable/user friendly for our users on the long run.

I do agree that the change on containerd side is unfortunate when I see how kind has been using containerd so far...

[Romain-Geissler-1A]

Something like an "extraCopiedFiles" (implemented with the oci-engine "cp" command obviously) with a top level API a bit like "extraMounts" would do, because it would look for "host files" on the client machine rather than on the host of the remote OCI engine, however this means implementing a kind of generic file manipulation API which seems out of the table. But from high level point of view, it would offer a "similar" API than "extraMounts" which exists already.

[BenTheElder]

Something like an "extraCopiedFiles" (implemented with the oci-engine "cp" command obviously) with a top level API a bit like "extraMounts" would do

This has to work across backends (podman, docker, nerdctl, and any future) but "cp" for these is not super well defined, and again, populating a file is pretty easy, we usually use docker exec cat to keep down the portable surface area and have better definition.

The real problem is mentioned above: if we do this, then file contents are trivial, what will become a pain is the file permissions, directory creation and permissions, and so on.

Instead if the user just wants to use the permissions from the host and handle all of that outside of kind, you can mount a directory with your containerd config using extraMounts and manipulate the files with standard tooling (e.g. a bash script creating the cluster, or checked-in config files in your repo)