search

loft-sh / vcluster

vCluster - Create fully functional virtual Kubernetes clusters - Each vcluster runs inside a namespace of the underlying k8s cluster. It's cheaper than creating separate full-blown clusters and it offers better multi-tenancy and isolation than regular namespaces.
https://www.vcluster.com
Apache License 2.0
6.26k stars 398 forks source link

Cannot deploy vcluster when running as root is not allowed #67

Closed rvdmei closed 2 years ago

rvdmei commented 3 years ago

I'm trying to deploy vcluster in a cluster that does not allow running containers as root, so the pods are not deploying with the current version of vcluster.

I have been checking the helm chart for an option to change this through values, but it looks like there isn't.

Is running as root an unavoidable requirement for vcluster or could this be fixed? (Running k3s as non-root is possible, although still experimental.)

FabianKramm commented 3 years ago

@rvdmei thanks for creating this issue! I don't think it is an unavoidable requirement and I think it should probably work without root, but we probably just have to try it. However, I'm not sure if coredns does support running as non root and this issue https://github.com/coredns/coredns/issues/1767 seems like they are not working on that anymore, but I could be wrong. This would be a problem as core dns is required for vcluster functioning correctly.

rvdmei commented 3 years ago

Looking at the issue it's not a coredns problem, but a simple permission problem. If you run coredns and listen on port 53 you either need root permissions or use libcap. Looks like they want to keep coredns platform independent which means no libcap. The fix is to not run coredns on port 53, but on a high port. My guess is that this is exactly what k3s is doing when running k3s rooltless.

FabianKramm commented 3 years ago

@rvdmei yes you are right, we can just add an option to vcluster to run in rootless mode or even do it by default if is has no side effects.

FabianKramm commented 3 years ago

After some investigation I don't think we can run rootless by default, because we need to share an emptyDir volume with the k3s container and have to access some of the k3s files, such as tls certificates, k3s kube config etc. By default k3s will create those files as root and restrict permissions to the owner. We probably would need to run as the same user as k3s is doing in rootless mode to be able to access those, however k3s throws a bunch of different errors if the flag --rootless is specified, which we would need to investigate first.

rvdmei commented 3 years ago

Rootless is still experimental for k3s, so I wouldn’t recommend running that as default anyway.

For emptyDir we have some restrictions too in the cluster, we must have a size set for OPA to allow the deployment. This is to prevent containers from using unlimited local storage or memory on the nodes, maybe the helm chart can support that too. Is a bit restrictive, bu knowing how many resources you are consuming is not a bad thing.