/var/lib/kube-router/kubeconfig does not regenerate when configmap changes are made

[BloodyIron]

Setting up kube-router for the first time in my rke2 cluster (managed by rancher + argocd).

I messed up the API endpoint declaration (protocol and port) in the configmap and when I corrected that in the config map the kube-router pods did not use the new changes at all, even after deleting them and all that.

Turns out the problem is that on each of my cluster nodes the file /var/lib/kube-router/kubeconfig does not get updated when said changes are made! It is only created upon initial provisioning of kube-router and does not seem to ever be modified or even removed when kube-router is de-provisioned!

To me this is a bug, and I think the area of code that would warrant improving is the initContainer "install-cni". However I don't know if that is re-triggered when the configmap is changed, but I do know that the steps checking for the presence of the kubeconfig file only checks that it is missing, not that it is different from "expected".

I had to go to each of my nodes and manually delete the kubeconfig file and re-provision kube-router for that file to reflect the new changes.

In my case, testing, this isn't that big of a problem. But for really large clusters this is a scaling problem. This also is something that could take a while for an admin to notice.

So I recommend that the kubeconfig file get regenerated on all nodes whenever relevant configmap changes are made. Can we get that please? :)

[aauren]

kube-router doesn't really do anything with this config, but it requires a valid kubeconfig file in order to run correctly. Specifically, since kube-router is the CNI that wires up the ClusterIP to the kube-apiserver it is not able to use the normal variables that are available to pods within a Kubernetes cluster because those variables will use the kube-apiserver ClusterIP which doesn't exist yet because kube-router has not set it up.

This presents a well known chicken and egg problem that is difficult for all CNIs to get around and all require some form of manual configuration to get around this. In our delivered daemonsets, we use the kubeconfig that is generated by most orchestrators for kube-proxy: https://github.com/cloudnativelabs/kube-router/blob/master/daemonset/kubeadm-kuberouter-all-features.yaml#L140-L145

If you're using this method of deployment, then restarting the pod should be all that is necessary to get a newer version of the ConfigMap to take effect. This is just a generic ConfigMap mount. So if that isn't working, then I would guess something else is very wrong in the cluster.

If you are presenting that file via some other mechanisms, then it would be a configuration for you to figure out in your cluster as it is beyond the scope of the project.

In regards to reloading an application when a ConfigMap is changed, this is a pretty well known problem space inside Kubernetes as most applications don't live load configuration files. So I would look at off-the-shelf type solutions that solve this already like reloader.

[BloodyIron]

I'm trying to use kube-router to get away from kube-proxy because it replaces the SourceIP for traffic in the cluster mode I'm using. So I don't know what the alternative method would be (from an automated perspective) to have that kubeconfig be regenerated on each node with changes.

Would not it be prudent for the init container to just check if the file-on-disk is different from the generated one, and then replace it with the latest one if such a difference exists?

"most applications don't live load configuration files"... uhhh HUP and other reloads have existed for a while, what are you on about here? nginx literally checks any configs when a reload happens for syntax errors and alerts if there's problems (I'm more describing nginx in a VM than in a container to be clear).

[aauren]

Sure, I'm just pointing out that, kube-router requires something like the configuration typically created for kube-proxy because they both require the same thing, which is a way of getting to the kube-apiserver BEFORE the ClusterIP for the kube-apiserver exists. You don't have to keep running kube-proxy if you don't want. Or you can generate your own config for kube-router in some way shape or form before kube-router starts.

kube-router doesn't generate or manage the file in /var/lib/kube-router/kube-config, that is expected to be provided by the orchestrator or operator of the cluster, so no it shouldn't do anything with the file other than read it.

In my comment, I said that most application's don't live load configuration files, not that there is no application that does. And I maintain that's still true, especially in the Kubernetes ecosystem. When you change a ConfigMap, I can't think of a single open source application that will re-read it in. That's why things like reloader exist. There is a huge difference between nginx and kube-router. If you restart nginx, you drop traffic, because the application actively sits in the data path. If you restart kube-router, the existing state of the network on the node continues to exist, because kube-router doesn't sit in the data path. While kube-router could implement HUP signals to reload config, you would still need something to send the signals, whereas there are things already built to reload the pod and at the end of the day it would be a lot of additional logic for zero gain.