Open edgan opened 2 years ago
xref kubernetes-sigs/aws-ebs-csi-driver#1274
It's a bug of aws-ebs-csi-driver helm chart. I've submitted a fix: https://github.com/kubernetes-sigs/aws-ebs-csi-driver/pull/1276
As a workaround you can try to add the following to the values:
sidecars:
nodeDriverRegistrar:
env:
- name: DRIVER_REG_SOCK_PATH
value: /var/lib/k0s/kubelet/plugins/ebs.csi.aws.com/csi.sock
As a workaround you can try to add the following to the values:
This isn't working for me. ebs-csi-node goes into a crashloop.
Sorry, I should have pointed out explicitly that you also need to specify the correct node.kubeletPath
. These values work for me:
sidecars:
nodeDriverRegistrar:
env:
- name: DRIVER_REG_SOCK_PATH
value: /var/lib/k0s/kubelet/plugins/ebs.csi.aws.com/csi.sock
node:
kubeletPath: /var/lib/k0s/kubelet
We've definitely seen some issues in our non-default location for kubelet data dir.
The main motivation for keeping everything under /var/lib/k0s
is driven by few different things:
when running k0s in containers/pods, we need to only worry about a single volume to be mounted
For changes like these, we need to also think hard on backwards compatibility. And especially in this case it's super difficult to actually change the kubelet directory. There's really no easy way to do an update for k0s, and thus for kubelet, in a way where we would not have to do a full reset first.
We do get that it's bit inconvenient in some cases, but as kubelet itself has a config option for the data-dir it uses, it's really pretty much always an upstream issue if those are not configurable. Yes, people make assumptions, but in this case those are kinda false assumptions. :) What we (the k0s team) can do here is to always help out in figuring out why something is not working with k0s and help making it work. Just like @makhov did now for the EBS CSI helm charts.
The issue is marked as stale since no activity has been recorded in 30 days
Closing the issue, since the fix for the EBS CSI helm chart was merged and we can't do more here. Feel free to reopen it as needed.
this is still a huge pain point
I agree. Had to debug an enterally day to discover what was the problem. For 1 of 2 Plugins, I solved it by adding a symbol link ln -s /var/lib/k0s/kubelet/ /var/lib/kubelet
(in the other case, I had no other choice than editing the k8s files).
I think they mean for us to "just" fork k0s and fix the path, which will have nothing but positive impacts on using it
@aronwolf90 @doctorpangloss If the symlink doesn't work, what about a bind mount? Could you maybe try mount --bind /var/lib/k0s/kubelet /var/lib/kubelet
and see if that fixes your issues?
Moreover, which plugins are you using that are lacking support for custom kubelet paths?
/cc #3508 which is similar, but the other way round.
@twz123 thanks for your time. In my case, it is an older version of https://github.com/hetznercloud/csi-driver (1.6) and yes, it can be fixed by downloading the yaml of csi-driver and adjust it (what is exactly what I did). The problem that I see here is, that I think that many others would have given up before finding the solution.
For my cluster, it is now fine as it is, but it is definitely a minus point when I have to consider what k8s distro I should recommend to others. This makes me a little sad because I really like the rest of k0s.
NOTE: mount --bind /var/lib/k0s/kubelet /var/lib/kubelet
does not work. In the logs I get failed to stage volume: mkdir /var/l ib/k0s/kubelet/plugins/kubernetes.io/csi/pv/pvc-54e61ed0-2bce-44cb-a980-1481b49a2b28/globalmount: no such file or director
.
The issue is marked as stale since no activity has been recorded in 30 days
The issue is marked as stale since no activity has been recorded in 30 days
I found that the "zero friction" moniker of k0s to be a bit misleading. Moving the kubelet folder is quite problematic. Here is how I solved it:
sudo mkdir -p /var/lib/k0s/kubelet/pods /var/lib/k0s/kubelet/plugins_registry /var/lib/k0s/kubelet/registration-dir /var/lib/k0s/kubelet/pods-mount-dir /var/lib/k0s/kubelet/plugins /var/lib/k0s/kubelet/device-plugins
sudo ln -s /var/lib/k0s/kubelet/pods /var/lib/kubelet/pods
sudo ln -s /var/lib/k0s/kubelet/plugins_registry /var/lib/kubelet/plugins_registry
sudo ln -s /var/lib/k0s/kubelet/registration-dir /var/lib/kubelet/registration-dir
sudo ln -s /var/lib/k0s/kubelet/pods-mount-dir /var/lib/kubelet/pods-mount-dir
sudo ln -s /var/lib/k0s/kubelet/plugins /var/lib/kubelet/plugins
sudo ln -s /var/lib/k0s/kubelet/device-plugins /var/lib/kubelet/device-plugins
curl -sSLf https://get.k0s.sh | sudo sh
...etc...
You have to run essentially the same series of commands on the control plane and all the workers before installing, otherwise some folders cannot be moved or overwritten due to locked .sock files. Maybe just symlinking /var/lib/kubelet -> /var/lib/k0s/kubelet would also work, I did not try that.
Before creating an issue, make sure you've checked the following:
Version
1.23.6
Platform
What happened?
aws-ebs-csi-driver out of the box failed to work with k0s.
Steps to reproduce
Expected behavior
PVC creation via aws-ebs-csi-driver works
Actual behavior
PVC creation via aws-ebs-csi-driver fails
Screenshots and logs
Jun 13 05:53:05 nexus-nexus-01 k0s[4738]: time="2022-06-13 05:53:05" level=info msg="E0613 05:53:05.391274 4928 nestedpendingoperations.go:335] Operation for \"{volumeName:kubernetes.io/csi/ebs.csi.aws.com^vol-0e2e37e9eecca4a86 podName: nodeName:}\" failed. No retries permitted until 2022-06-13 05:54:09.391247943 +0000 UTC m=+1996.540517455 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume \"pvc-21851630-5134-4690-9807-91e57ef51902\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e2e37e9eecca4a86\") pod \"nexus-repo-nexus-repository-manager-5b5ff457c5-n5k9l\" (UID: \"753df29d-d7de-4d68-bd24-1538e1b1eafc\") : applyFSGroup failed for vol vol-0e2e37e9eecca4a86: lstat /var/lib/k0s/kubelet/pods/753df29d-d7de-4d68-bd24-1538e1b1eafc/volumes/kubernetes.io~csi/pvc-21851630-5134-4690-9807-91e57ef51902/mount: no such file or directory" component=kubelet
Additional context
The issue is that
aws-ebs-csi-driver
expects things in/var/lib/kubelet
not/var/lib/k0s/kubelet
.Things tried:
/var/lib/kubelet
to/var/lib/k0s/kubelet
/var/lib/k0s/kubelet
to/var/lib/kubelet
/var/lib/k0s/kubelet
to/var/lib/kubelet
aws-ebs-csi-driver
helm chart by hand to use/var/lib/k0s/kubelet
Results:
In the process of tracking down this issue I ran across multiple previous
k0s
issues and multiple CSI driver issues across GitHub projects that all point to many Kubernetes projects assume/var/lib/kubelet
. The ideal would be for it to always be configurable, but the reality is that people do assume. The ultimate issue is thatk0s
is breaking compatibility with a lot of other projects by changing the directory from/var/lib/kubelet
to/var/lib/k0s/kubelet
.