Move to using the kubelet config file for config instead of cmd line args

[davidnuzik]

Migrate to using kubelet config file for configuring kubelets rather than the command line args we are using today since upstream is moving away from using args.

EDIT 2024-06-28: Picking this up for 1.30+, now that Drop-in directory for kubelet configuration files is in beta and enabled by default.

Ref: https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/#kubelet-configuration-merging-order

Upstream order is:

1. Feature gates specified over the command line (lowest precedence).
2. The kubelet configuration.
3. Drop-in configuration files, according to sort order.
4. Command line arguments excluding feature gates (highest precedence).

Our order will be:

1. K3s defaults via drop-in at data-dir/agent/etc/kubelet.conf.d/00-k3s-defaults.conf
2. User-specified --config copied to drop-in at data-dir/agent/etc/kubelet.conf.d/10-cli-config.conf
3. User-specified --config-dir drop-ins copied to data-dir/agent/etc/kubelet.conf.d/20-cli-config-dir/
4. User-specified --kubelet-arg command line arguments (passed through using current logic)

Given:

1. K3s command -line args: --kubelet-arg=config=/etc/rancher/k3s/kubelet.conf --kubelet-arg=config-dir=/etc/rancher/k3s/kubelet.conf.d --kubelet-arg=image-gc-high-threshold=100
2. Files /etc/rancher/k3s/kubelet.conf and /etc/rancher/k3s/kubelet.conf.d/extra-config.conf

K3s will do the following:

1. Write default k3s configuration at /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/00-k3s-defaults.conf
2. Copy /etc/rancher/k3s/kubelet.conf to /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/10-cli-config.conf
3. Copy /etc/rancher/k3s/kubelet.conf.d/extra-config.conf to /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/20-cli-config-dir/extra-config.conf
4. Remove the --config and --config-dir flags from the user-provided kubelet-arg list
5. Set --config-dir=/var/lib/rancher/k3s/agent/etc/kubelet.conf.d as default kubelet args
6. Append remaining kubelet-arg values (--image-gc-high-threshold=100) to default args, and pass through as kubelet command line flags

Notes:

• 1.28 and 1.29 also support --config-dir if you set the KUBELET_CONFIG_DROPIN_DIR_ALPHA env var, but since we generally try to avoid using alpha features, I would probably prefer to make this 1.30+ only.

[dereknola]

Working on this. Developing a hybrid approach where if no user provides a custom config, we will use a config.

[brandond]

We should do this for all the components, not just the Kubelet. Ideally we'd have some way to translate custom args (kubelet-arg, kube-apiserver-arg, so on) work with config files since Rancher and many other tools still use those to configure things and probably won't switch over to config files any time soon.

[dereknola]

@brandond Currently only kubelet allows the use of configuration files. As you can see from https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ kubelet describes in multiple places that CLI is being deprecated and provides a section on the new configuration. However for kube-apiserver no such section on deprecation, or even the option to configure via config is provided. Same for kube-controller and other K8s components

[brandond]

I'm trying to find the KEP, but at one point they were talking about doing away with CLI flags entirely, in favor of typed configuration files for everything. The kubelet was the first thing they tackled, and added all the deprecation warnings to the flags, but since then they have sort of abandoned the push - they've not actually worked towards removing the flags, or marking flags deprecated on other components. They instead seem to be pushing back on adding any new CLI flags.

But yeah you're 100% right, since they haven't moved forward with config files for the other components, we don't need to worry about it yet.

[brandond]

Picking this up with some ideas of how to use drop-in files to enhance the ability to override our defaults.

[fmunteanu]

@brandond what’s the K3s release target for this implementation?

[brandond]

We don't have a hard target. Next months release would be nice.