search

k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
28.23k stars 2.36k forks source link

Move to using the kubelet config file for config instead of cmd line args #2116

Open davidnuzik opened 4 years ago

davidnuzik commented 4 years ago

Migrate to using kubelet config file for configuring kubelets rather than the command line args we are using today since upstream is moving away from using args.

EDIT 2024-06-28: Picking this up for 1.30+, now that Drop-in directory for kubelet configuration files is in beta and enabled by default.

Ref: https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/#kubelet-configuration-merging-order

Upstream order is:

  1. Feature gates specified over the command line (lowest precedence).
  2. The kubelet configuration.
  3. Drop-in configuration files, according to sort order.
  4. Command line arguments excluding feature gates (highest precedence).

Our order will be:

  1. K3s defaults via drop-in at data-dir/agent/etc/kubelet.conf.d/00-k3s-defaults.conf
  2. User-specified --config copied to drop-in at data-dir/agent/etc/kubelet.conf.d/10-cli-config.conf
  3. User-specified --config-dir drop-ins copied to data-dir/agent/etc/kubelet.conf.d/20-cli-config-dir/
  4. User-specified --kubelet-arg command line arguments (passed through using current logic)

Given:

  1. K3s command -line args: --kubelet-arg=config=/etc/rancher/k3s/kubelet.conf --kubelet-arg=config-dir=/etc/rancher/k3s/kubelet.conf.d --kubelet-arg=image-gc-high-threshold=100
  2. Files /etc/rancher/k3s/kubelet.conf and /etc/rancher/k3s/kubelet.conf.d/extra-config.conf

K3s will do the following:

  1. Write default k3s configuration at /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/00-k3s-defaults.conf
  2. Copy /etc/rancher/k3s/kubelet.conf to /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/10-cli-config.conf
  3. Copy /etc/rancher/k3s/kubelet.conf.d/extra-config.conf to /var/lib/rancher/k3s/agent/etc/kubelet.conf.d/20-cli-config-dir/extra-config.conf
  4. Remove the --config and --config-dir flags from the user-provided kubelet-arg list
  5. Set --config-dir=/var/lib/rancher/k3s/agent/etc/kubelet.conf.d as default kubelet args
  6. Append remaining kubelet-arg values (--image-gc-high-threshold=100) to default args, and pass through as kubelet command line flags

Notes:

dereknola commented 1 year ago

Working on this. Developing a hybrid approach where if no user provides a custom config, we will use a config.

brandond commented 1 year ago

We should do this for all the components, not just the Kubelet. Ideally we'd have some way to translate custom args (kubelet-arg, kube-apiserver-arg, so on) work with config files since Rancher and many other tools still use those to configure things and probably won't switch over to config files any time soon.

dereknola commented 1 year ago

@brandond Currently only kubelet allows the use of configuration files. As you can see from https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/ kubelet describes in multiple places that CLI is being deprecated and provides a section on the new configuration. However for kube-apiserver no such section on deprecation, or even the option to configure via config is provided. Same for kube-controller and other K8s components

brandond commented 1 year ago

I'm trying to find the KEP, but at one point they were talking about doing away with CLI flags entirely, in favor of typed configuration files for everything. The kubelet was the first thing they tackled, and added all the deprecation warnings to the flags, but since then they have sort of abandoned the push - they've not actually worked towards removing the flags, or marking flags deprecated on other components. They instead seem to be pushing back on adding any new CLI flags.

But yeah you're 100% right, since they haven't moved forward with config files for the other components, we don't need to worry about it yet.

brandond commented 5 months ago

Picking this up with some ideas of how to use drop-in files to enhance the ability to override our defaults.

fmunteanu commented 3 months ago

@brandond what’s the K3s release target for this implementation?

brandond commented 3 months ago

We don't have a hard target. Next months release would be nice.

dexmanley commented 2 months ago

@brandond is there plans yet to add this back into an upcoming release?

brandond commented 2 months ago

There is a bug in the dropin config loading that is only going to be fixed in v1.32; upstream is not backporting the fix because it's not a regression, its just a bug in a beta feature. I think we should not use this feature on releases that do not have the fix.

For that reason I think I will schedule this for k3s v1.32.0+k3s1

fmunteanu commented 2 months ago

There is a bug in the dropin config loading that is only going to be fixed in v1.32; upstream is not backporting the fix because it's not a regression, its just a bug in a beta feature. I think we should not use this feature on releases that do not have the fix.

For that reason I think I will schedule this for k3s v1.32.0+k3s1

Thank you for the update, @brandond, should we create a new issue to track this change 1.32? Not sure what is your development cycle.

brandond commented 2 months ago

no? this issue is already in the milestone.