kubelet unable to start

tpaul1611 commented 2 weeks ago

Version

Karpenter Version: v0.5.1

Kubernetes Version: v1.30.3

Expected Behavior

I have duplicated an already working selfhosted karpenter setup. There are no apparent differences in the clusters or karpenter configuration. It should work.

Actual Behavior

Karpenter is creating VMs but they never join the cluster, because their kubelet is unable to start. The issue persists even after a cluster recreation.

Steps to Reproduce the Problem

Unsure, since the same setup works on other clusters.

Resource Specs and Logs

These are the logs of the kubelet on the VM:

Sep 18 08:06:32 aks-spot-6wxdv systemd[1]: Starting Kubelet... Sep 18 08:06:32 aks-spot-6wxdv bash[3020]: Primary NIC IP: 10.16.0.112 Sep 18 08:06:32 aks-spot-6wxdv bash[3020]: Insert IMDS restriction rule to mangle table: Sep 18 08:06:32 aks-spot-6wxdv bash[3020]: Enable IMDS restriction: Sep 18 08:06:32 aks-spot-6wxdv bash[3020]: One or more required variables are not set, exiting... Sep 18 08:06:32 aks-spot-6wxdv systemd[1]: kubelet.service: Control process exited, code=exited, status=1/FAILURE Sep 18 08:06:32 aks-spot-6wxdv systemd[1]: kubelet.service: Failed with result 'exit-code'. Sep 18 08:06:32 aks-spot-6wxdv systemd[1]: Failed to start Kubelet. Sep 18 08:06:34 aks-spot-6wxdv systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 1. Sep 18 08:06:34 aks-spot-6wxdv systemd[1]: Stopped Kubelet. Sep 18 08:06:34 aks-spot-6wxdv systemd[1]: Starting Kubelet... Sep 18 08:06:34 aks-spot-6wxdv bash[3086]: Primary NIC IP: 10.16.0.112 Sep 18 08:06:34 aks-spot-6wxdv bash[3086]: Insert IMDS restriction rule to mangle table: Sep 18 08:06:34 aks-spot-6wxdv bash[3086]: Enable IMDS restriction: Sep 18 08:06:34 aks-spot-6wxdv bash[3086]: One or more required variables are not set, exiting... Sep 18 08:06:34 aks-spot-6wxdv systemd[1]: kubelet.service: Control process exited, code=exited, status=1/FAILURE Sep 18 08:06:34 aks-spot-6wxdv systemd[1]: kubelet.service: Failed with result 'exit-code'. Sep 18 08:06:34 aks-spot-6wxdv systemd[1]: Failed to start Kubelet. Sep 18 08:06:36 aks-spot-6wxdv systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 2. Sep 18 08:06:36 aks-spot-6wxdv systemd[1]: Stopped Kubelet. Sep 18 08:06:36 aks-spot-6wxdv systemd[1]: Starting Kubelet... Sep 18 08:06:36 aks-spot-6wxdv bash[3374]: Primary NIC IP: 10.16.0.112 Sep 18 08:06:36 aks-spot-6wxdv bash[3374]: Insert IMDS restriction rule to mangle table: Sep 18 08:06:36 aks-spot-6wxdv bash[3374]: Enable IMDS restriction: Sep 18 08:06:36 aks-spot-6wxdv bash[3374]: One or more required variables are not set, exiting... Sep 18 08:06:36 aks-spot-6wxdv systemd[1]: kubelet.service: Control process exited, code=exited, status=1/FAILURE Sep 18 08:06:36 aks-spot-6wxdv systemd[1]: kubelet.service: Failed with result 'exit-code'. Sep 18 08:06:36 aks-spot-6wxdv systemd[1]: Failed to start Kubelet. Sep 18 08:06:37 aks-spot-6wxdv systemd[1]: Stopped Kubelet.

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

tpaul1611 commented 2 weeks ago

aparently this was fixed by karpenter 0.5.4

tallaxes commented 2 weeks ago

Yes, fixed by #484 in 0.5.4 release

Azure / karpenter-provider-azure