Timed out waiting for the condition

[clincha]

Running the following command on the Kuberentes master causes an error:

kubeadm token create --print-join-command

fatal: [bri-master-1]: FAILED! => {"changed": false, "cmd": ["kubeadm", "token", "create", "--print-join-command"], "delta": "0:01:15.066834", "end": "2023-06-10 16:31:49.855806", "msg": "non-zero return code", "rc": 1, "start": "2023-06-10 16:30:34.788972", "stderr": "timed out waiting for the condition\nTo see the stack trace of this error execute with --v=5 or higher", "stderr_lines": ["timed out waiting for the condition", "To see the stack trace of this error execute with --v=5 or higher"], "stdout": "", "stdout_lines": []}

Increasing the verbosity (kubeadm token create --print-join-command --v=6) revealed another error:

I0610 16:37:44.528130   91993 round_trippers.go:553] POST https://192.168.1.20:6443/api/v1/namespaces/kube-system/secrets?timeout=10s 500 Internal Server Error in 1 milliseconds

[clincha]

Increasing the verbosity level to 9 exposed a deeper error

0610 21:40:24.616644  171225 round_trippers.go:553] POST https://192.168.1.20:6443/api/v1/namespaces/kube-system/secrets?timeout=10s 500 Internal Server Error in 1 milliseconds
I0610 21:40:24.616675  171225 round_trippers.go:570] HTTP Statistics: GetConnection 0 ms ServerProcessing 0 ms Duration 1 ms
I0610 21:40:24.616684  171225 round_trippers.go:577] Response Headers:
I0610 21:40:24.616689  171225 round_trippers.go:580]     X-Kubernetes-Pf-Flowschema-Uid: ddbf6f0f-071a-4ba9-a930-2612fd0123d0
I0610 21:40:24.616696  171225 round_trippers.go:580]     X-Kubernetes-Pf-Prioritylevel-Uid: 261b1fb2-3532-4155-8047-53029f051ca1
I0610 21:40:24.616700  171225 round_trippers.go:580]     Content-Length: 227
I0610 21:40:24.616706  171225 round_trippers.go:580]     Date: Sat, 10 Jun 2023 20:40:24 GMT
I0610 21:40:24.616711  171225 round_trippers.go:580]     Audit-Id: 8bc153ba-632c-4bf7-ab35-4a1f226c1eef
I0610 21:40:24.616717  171225 round_trippers.go:580]     Cache-Control: no-cache, private
I0610 21:40:24.616723  171225 round_trippers.go:580]     Content-Type: application/json
I0610 21:40:24.616749  171225 request.go:1171] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Get \"https://[::1]:6443/api/v1/namespaces/kube-system/resourcequotas\": dial tcp [::1]:6443: connect: cannot assign requested address","code":500}

In that error I see an ipv6 address that it's trying to resolve to. I can enable ipv6 in sysctl.conf and reboot to get ipv6 back up. Then I can test again.

vi /etc/sysctl.conf

Comment out this line:

net.ipv6.conf.all.disable_ipv6 = 1

Reboot the server and test again... it works. I'll remove the code that disables ipv6. I can't remember why I put it in.

[clincha]

I removed the ipv6 code and it hasn't caused issues yet.