k3s service failed to start

Environmental Info: K3s Version:

k3s version v1.30.6+k3s1 (1829eaae)
go version go1.22.8

Node(s) CPU architecture, OS, and Version:

Linux main 6.11.0-9-generic #9-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 14 13:19:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Linux gpu 6.11.0-9-generic #9-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 14 13:19:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Linux pi 6.11.0-1004-raspi #4-Ubuntu SMP PREEMPT_DYNAMIC Fri Sep 27 22:25:34 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux

Cluster Configuration: 1 Server, 2 Agents

Describe the bug: After a reboot, the server node won't start.

2024-11-25T16:29:42.897362+00:00 main systemd[1]: k3s.service: Scheduled restart job, restart counter is at 201.
2024-11-25T16:29:42.899480+00:00 main systemd[1]: Starting k3s.service - Lightweight Kubernetes...
2024-11-25T16:29:43.066326+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Starting k3s v1.30.6+k3s1 (1829eaae)"
2024-11-25T16:29:43.067185+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Configuring sqlite3 database connection pooling: maxIdleConns=2, maxOpenConns=0, connMaxLifetime=0s"
2024-11-25T16:29:43.067270+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Configuring database table schema and indexes, this may take a moment..."
2024-11-25T16:29:43.067454+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Database tables and indexes are up to date"
2024-11-25T16:29:43.071738+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Kine available at unix://kine.sock"
2024-11-25T16:29:43.076193+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=info msg="Saving cluster bootstrap data to datastore"
2024-11-25T16:29:43.076949+00:00 main k3s[6509]: time="2024-11-25T16:29:43Z" level=fatal msg="starting kubernetes: preparing server: failed to normalize server token; must be in format K10<CA-HASH>::<USERNAME>:<PASSWORD> or <PASSWORD>"

Steps To Reproduce: I am new to Kubernetes and k3s, so I tried a lot. I have reinstalled k3s multiple times today, did many upgrades and configuring. The last thing I was doing was installing longhorn and applying some cert-manager configuration. Then I installed the longhorn drivers and added a default certificate. After a reboot, the server node wouldn't start.

The /var/lib/rancher/k3s/server/node-token was empty. Since I joined all the other servers today, I still have the original contents. Like in #5345 I tried to remove the file, but k3s does not want to start. Also, adding the contents that I used to join the other nodes didn't work either.

Expected behavior:

k3s should start normal.

Actual behavior:

k3s service exits with a fatal error

Additional context / logs:

Scrolling through the logs, it seems like I installed k3s and k3s-agent on the same machine. Stopping k3s-agent and trying to start k3s did not work. Is there something to be overwritten, or can I still salvage this install? Thanks for your help.

k3s-io / k3s

k3s service failed to start #11366