Renew API cert with added --tls-san

[gazzer82]

I have a 3 master k3s cluster setup on Ubuntu 20, running etcd.

I need to update the api cert to add an additional floating IP to the cert, so I've updated the k3s.service file on master 1, adding a second --tls-san option for that IP.

So it now looks like this:

ExecStart=/usr/local/bin/k3s \
    server --disable servicelb --disable traefik \
        '--cluster-init' \
        '--tls-san' \
        '11.11.11.11' \
        '--tls-san' \
        '12.12.12.12' \

Master 2/3 have service files that look like this, so I have made no changes to them.

ExecStart=/usr/local/bin/k3s \
    server --disable servicelb --disable traefik \
        '--server' \
        'https://10.10.10.10:6443' \

I have restarted the K3S service on all systems, rebooted master 1, and fully drained master 2 and master 3, but the API cert doesn't seem to be updating with the additional IP address?

If there a way to manually force the certificate to be rotated with the new config, or steps I need to take to force the rotation and then updating of the cert on the management pane for all masters?

I have done some googling, but haven't managed to find anything, apologies if I've missed the steps somewhere!

Thanks

Gareth

[brandond]

You should see additional DNS or IP SAN entries show up on the port 6443 listener after you have made a request to that IP or hostname.

[gazzer82]

You should see additional DNS or IP SAN entries show up on the port 6443 listener after you have made a request to that IP or hostname.

I’m assuming that’s after updating the —tls-san parameter and restarting the service? When I was testing that earlier that didn’t seem to be working, subsequent requests where reporting the same inital IP addresses?

[brandond]

Are you getting an error when accessing it at the floating IP address? Have you added that arg to all the nodes that might host the floating IP? The cert is a bit odd in that it is dynamically updated on-demand to add valid addresses so it's not surprising that you might not see what you want right away if you're not actually hitting it with traffic at that address yet.

[kidhasmoxy]

Running into the same issue with the same setup (cluster with embedded etcd) and adding the load balancer after.

Have added the tls parameter to the service setup of all 3 servers, restarted the service, attempted to use kubectl and got the tls error. Get the error every time I try to access via the load balancer IP.

Will it only trigger if I add a worker to the cluster using the load balancer ip?

[ambis]

I had the same problem initially with v1.20.4+k3s1, using K3S_CONFIG_FILE which had the tls-san spesified.

When running the installer script after changes, it just said No change detected so skipping service start. So I restarted each k3s instance manually with systemctl restart k3s.

It took some minutes after that when kubectl started to work using the new hostnames without tls error.

Maybe the change detection algorithm in the installer needs a bit of TLC, to check if K3S_CONFIG_FILE has any changes in it?

It could also be, that k3s tracks chages to the config file and the whole restart process was unnecessary...

[brandond]

That is correct. If you are relying on the install script to restart K3s for you, you should be aware that it only detects changes to the systemd unit. If you change the environment or config file, you should manually restart the service.

In general I wouldn't rely on the install script as a general-purpose utility for changing configuration. If that's all you need to do, just edit the systemd unit, environment file, or config file, and then reload/restart the service.

[stale[bot]]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 180 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[sourcehawk]

Can anyone explain what needs to be changed and where in order to update the tls-san options? Or maybe link some useful documentation that I can read upon to understand how to configure it?

I just want to add a domain to the tls-san but I don't have a clue how to do it. I can only find half a sentence of context for this functionality in the k3s docs.

[FischerLGLN]

I've found the solution here, you just have to renew the dns cache. I could execute this on the master node:

export MY_LB_IP=<IP>
curl -vk --resolve $MY_LB_IP:6443:127.0.0.1  https://$MY_LB_IP:6443/ping

No changes to the external/client kubeconfig are needed after this!

[svenakela]

This is my solution.

Make a /etc/rancher/k3s/config.yaml file with your settings. I have:

write-kubeconfig-mode: "0644"
token: hoolabaloosecrettoken
tls-san:
  - 192.168.10.4
  - 192.168.10.10

where 10.4 is the IP of the server NIC and 10.10 is the virtual IP I use to access the cluster. For me that IP was the issue as I updated the load balancer.

Delete the defunct certificate and restart the server:

node1$ kubectl -n kube-system delete secrets/k3s-serving
node1$ sudo mv /var/lib/rancher/k3s/server/tls/dynamic-cert.json /tmp/dynamic-cert.json
node1$ sudo systemctl restart k3s

I am not sure it is needed but for redundancy I have the config (with the first IP altered) on each node in the cluster and they are all restarted after the IP update. Now the new IP is working with kubectl.