Running Microk8s in an unprivileged LXD container

[matjazp]

Official docs for running Microk8s inside LXD containers still define microk8s LXD profile that runs it in as a privileged container and also disables a bunch of other security related features.

What is missing to run Microk8s in a regular, unprivileged container? I'm running Microk8s in a VM, but would love to have an option for running it in (unprivileged) containers. You can already run Docker like that, so nested container runtimes are probably not the main issue?

[ktsakalozos]

The profile we have for LXD is very permissive so as to not block any workloads users may want to run in Kubernetes. Users with well defined workloads can start with the wide open LXD profile and try to make it less permissive allowing only the capabilities their workload requires.

[sashati]

I face with the same issue. As you know the nvidia.runtime just works in unprivileged mode. Then if we want to utilize GPU on MicroK8s, there is indeed no way. Does anyone find a solution for MicroK8s on unprivileged LXC?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[neoaggelos]

not stale, not completed

[dalbani]

Coming here from https://github.com/lxc/lxd/issues/4184, my understanding is that having the ability to run MicroK8s in an unprivileged container would open up possibilities with ZFS zones.

@stgraber wrote in particular:

Because the zfs zone stuff is based on the user namespace, I don't expect it to ever work with security.privileged=true as that turns off the user namespace. You'd need to get microk8s working properly without security.privileged=true for any of this to work there.

I'm not sure I fully understood your comment, @ktsakalozos. You talked about (specific) workloads, but what about MicroK8s itself first. Can it (be made to) work in an unprivileged container?

[dalbani]

I'm not sure if it's relevant to this discussion, but I stumbled upon this concept of "rootless mode" in Kubernetes: https://kubernetes.io/docs/tasks/administer-cluster/kubelet-in-userns/

This document describes how to run Kubernetes Node components such as kubelet, CRI, OCI, and CNI without root privileges, by using a user namespace.

Which relies on the KubeletInUserNamespace feature gate from what I can read.

And while we're talking about MicroK8s here, I found some similar discussion regarding K3s: https://github.com/k3s-io/k3s/issues/4249.

KubeletInUserNamespace is not set in unprivileged LXD containers when k3s is run as root

And, from the discussion on this page, it looks like K3s does work in an unprivileged LXD container thanks to this mode. Or am I understanding it incorrectly?

If my interpretation is correct, could the same capability be built in MicroK8s?

[ktsakalozos]

@dalbani this is an interesting feature/setup configuration that up to this point we have not looked into it.

[dalbani]

Thanks @ktsakalozos for the feedback 👍 For the record, a competitor product like K3s offers an (experimental) rootless mode: https://docs.k3s.io/advanced#running-rootless-servers-experimental

[matjazp]

Rootless mode can be a welcomed addition, but it (currently) has many limitations (see docs). I would still prefer a regular K8s.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[dalbani]

/unstale

[Pictor13]

What's the status?
Is it mandatory to use a privileged Docker context, for now, in order to run MicroK8s?
Is it a requirement?

~Installing recent Docker on Linux releases seems to use rootless context by default.~