VannTen
commented
1 day ago That drop-in looks out of place. Kubespray does not use distro systemd unit files AFAIK, so this one might be interfering with the kubespray settings in unexpected ways.
Closed hufhend closed 1 day ago
VannTen
commented
1 day ago That drop-in looks out of place. Kubespray does not use distro systemd unit files AFAIK, so this one might be interfering with the kubespray settings in unexpected ways.
hufhend
commented
1 day ago Maybe he's not using a distro, but creating his own. Either way, he's using them, here I see in kubespray:
- name: Enable kubelet
service:
name: kubelet
enabled: true
state: startedThat's not the point. The systemd service is probably from kubespray, since it's in /etc as your logs show. But the override is in /usr/lib which makes me think it's part of a distro package. The combination of those two might not be something working.
Anyway, systemctl cat kubelet and journalctl -u kubelet might help you to diagnose the exact problem.
hufhend
commented
1 day ago I checked it, it's exactly the same as on Ubuntu 22.04 where it works:
# /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet Server
Documentation=https://github.com/GoogleCloudPlatform/kubernetes
After=containerd.service
Wants=containerd.service
[Service]
EnvironmentFile=-/etc/kubernetes/kubelet.env
ExecStart=/usr/local/bin/kubelet \
$KUBE_LOGTOSTDERR \
$KUBE_LOG_LEVEL \
$KUBELET_API_SERVER \
$KUBELET_ADDRESS \
$KUBELET_PORT \
$KUBELET_HOSTNAME \
$KUBELET_ARGS \
$DOCKER_SOCKET \
$KUBELET_NETWORK_PLUGIN \
$KUBELET_VOLUME_PLUGIN \
$KUBELET_CLOUDPROVIDER
Restart=always
RestartSec=10s
[Install]
WantedBy=multi-user.target
# /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf
# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGSExecStart= ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS
As I was saying. This use kubelet from the distro package instead of the one provided by kubespray.
Mixing configuration from kubespray and other sources for the kubelet is not supported, and won't be. Try removing the packages owning that override and see if the issue persist.
If it does, then you should open a new bug report
/kind support /close
k8s-ci-robot
commented
1 day ago @VannTen: Closing this issue.
hufhend
commented
1 day ago No, there is no distribution version installed.
apt-cache policy kubelet
kubelet:
Installed: (none)
Candidate: 1.30.6-1.1
Version table:
1.30.6-1.1 500
500 https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
1.30.5-1.1 500
500 https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
1.30.4-1.1 500
500 https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
1.30.3-1.1 500
500 https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
1.30.2-1.1 500
500 https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
1.30.1-1.1 500
500 https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
1.30.0-1.1 500
500 https://pkgs.k8s.io/core:/stable:/v1.30/deb Packages
What happened?
What did you expect to happen?
I would expect it to run and have the right variables, like here on Ubuntu 22.04:
How can we reproduce it (as minimally and precisely as possible)?
For me it only shows up on Ubuntu 24.04 and Armbian Linux v24.11 rolling for Orange Pi Zero3, I assume the problem is related to these distributions.
OS
and
Version of Ansible
Version of Python
Python 3.12.3
Version of Kubespray (commit)
4577ee4a5
Network plugin used
calico
Full inventory with variables
n-pro04-dc3 | SUCCESS => { "hostvars[kube_node]": "VARIABLE IS NOT DEFINED!" }
Command used to invoke ansible
ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root cluster.yml -b -l n-pro04-dc3
Output of ansible run
Surprisingly fine and bug-free
Anything else we need to know
It used to load and run normally, now the kubelet crashes right after the kubespray runs out. Something must have changed between the distribution and kubespray, anyway the kubelet doesn't have its variables, although the file with them is there /usr/lib/systemd/system/kubelet.service.d/10-kubeadm.conf