Open davidnuzik opened 4 years ago
Working on this. Developing a hybrid approach where if no user provides a custom config, we will use a config.
We should do this for all the components, not just the Kubelet. Ideally we'd have some way to translate custom args (kubelet-arg, kube-apiserver-arg, so on) work with config files since Rancher and many other tools still use those to configure things and probably won't switch over to config files any time soon.
@brandond Currently only kubelet allows the use of configuration files. As you can see from https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ kubelet describes in multiple places that CLI is being deprecated and provides a section on the new configuration. However for kube-apiserver no such section on deprecation, or even the option to configure via config is provided. Same for kube-controller and other K8s components
I'm trying to find the KEP, but at one point they were talking about doing away with CLI flags entirely, in favor of typed configuration files for everything. The kubelet was the first thing they tackled, and added all the deprecation warnings to the flags, but since then they have sort of abandoned the push - they've not actually worked towards removing the flags, or marking flags deprecated on other components. They instead seem to be pushing back on adding any new CLI flags.
But yeah you're 100% right, since they haven't moved forward with config files for the other components, we don't need to worry about it yet.
Picking this up with some ideas of how to use drop-in files to enhance the ability to override our defaults.
@brandond what’s the K3s release target for this implementation?
We don't have a hard target. Next months release would be nice.
Migrate to using kubelet config file for configuring kubelets rather than the command line args we are using today since upstream is moving away from using args.
EDIT 2024-06-28: Picking this up for 1.30+, now that
Drop-in directory for kubelet configuration files
is in beta and enabled by default.Ref: https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/#kubelet-configuration-merging-order
Upstream order is:
Our order will be:
Given:
--kubelet-arg=config=/etc/rancher/k3s/kubelet.conf --kubelet-arg=config-dir=/etc/rancher/k3s/kubelet.conf.d --kubelet-arg=image-gc-high-threshold=100
/etc/rancher/k3s/kubelet.conf
and/etc/rancher/k3s/kubelet.conf.d/extra-config.conf
K3s will do the following:
/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/00-k3s-defaults.conf
/etc/rancher/k3s/kubelet.conf
to/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/10-cli-config.conf
/etc/rancher/k3s/kubelet.conf.d/extra-config.conf
to/var/lib/rancher/k3s/agent/etc/kubelet.conf.d/20-cli-config-dir/extra-config.conf
--config
and--config-dir
flags from the user-provided kubelet-arg list--config-dir=/var/lib/rancher/k3s/agent/etc/kubelet.conf.d
as default kubelet args--image-gc-high-threshold=100
) to default args, and pass through as kubelet command line flagsNotes:
--config-dir
if you set theKUBELET_CONFIG_DROPIN_DIR_ALPHA
env var, but since we generally try to avoid using alpha features, I would probably prefer to make this 1.30+ only.