Expose CFS quota kubelet option

[johananl]

In some cases it may be desirable to tweak the CFS quota kubelet option which affects CPU throttling of containers.

There seem to be at least two ways to specify this option:

1. Using the --cpu-cfs-quota-period kubelet CLI argument.
2. Using the kubelet config file specified via the --config kubelet CLI argument.

The former is marked as deprecated in the docs. The latter is in beta at time of writing, however we are already using the kubelet config file for something else:

https://github.com/kinvolk/lokomotive/blob/506547593fca959f9dc26b03720bbee74a56e081/assets/terraform-modules/packet/flatcar-linux/kubernetes/workers/cl/worker.yaml.tmpl#L111

Things to consider:

• We may want to use this opportunity to allow arbitrary kubelet flags to be specified in the config.
• Ideally we should be able to leverage the fact that we use self-hosted kubelets to avoid having to replace nodes when making changes (but then what about the native kubelet?).
• There seems to be a way to configure kubelets dynamically. This could be relevant here.

[invidian]

Ideally we should be able to leverage the fact that we use self-hosted kubelets to avoid having to replace nodes when making changes (but then what about the native kubelet?).

We could make DaemonSet to write kubeconfig file to the host. However, this won't allow different configs per worker pool.