search

rootless-containers / usernetes

Kubernetes without the root privileges
https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2033-kubelet-in-userns-aka-rootless
Apache License 2.0
853 stars 58 forks source link

wip: usernetes in lima #308

Closed vsoch closed 6 months ago

vsoch commented 7 months ago

Hey @AkihiroSuda ! I gave your suggestion a try and was able to get usernetes working in Lima (SO much easier!) https://github.com/converged-computing/usernetes-lima. I wanted to ask for your advice for how to automate some of the usernetes setup itself. Right now, since I need to run the setup as the user, I basically have the files mounted or available in the present working directory. I'm running into two issues:

Provision scripts

I bootstrapped the rockylinux config, and tried to customize the startup procedure, but it seems like this portion isn't running: https://github.com/converged-computing/usernetes-lima/blob/c91093a48ca59e4d28161895c31cfea078ca484c/usernetes.yaml#L35-L68. I'm going to read up more on these scripts https://docs.rancherdesktop.io/how-to-guides/provisioning-scripts/ and try to debug. Ideally I can run the contents of rootless.sh (minimally) in that section, and then have separate yaml files for a control plane and worker nodes.

I want to open this issue for discussion and I can post updates, if that is ok. Happy weekend!

vsoch commented 7 months ago

okay first part (root run commands) seems to be mostly working. The trick was to add a probe that watches for the kubectl binary (the last block) to exist. https://github.com/converged-computing/usernetes-lima/pull/1. Going to see if there is a way to run something in userspace next.

afbjorklund commented 7 months ago

I made an example based on the existing k8s.yaml, which uses the standard Ubuntu LTS instead of something Red Hat...

vsoch commented 7 months ago

Awesome! I can try that out. But rockylinux isn't Red Hat - what am I missing?

Also I see how to run something in user space - I didn't know how to do that, and I'll update my rockylinux variant to do the same. Thank you!

vsoch commented 7 months ago

@afbjorklund question for you - I noticed that without the probes, often my provision steps wouldn't complete. I also wasn't sure if they are run in serial (of expected to work independently in parallel). Are the provision steps run in a sequence, and so the probes should generally sanity check across different provision steps? Are there any gotchas?

vsoch commented 7 months ago

And follow up question - how do you decide what to scope to different provision blocks (vs having one large one)?

afbjorklund commented 7 months ago

But rockylinux isn't Red Hat - what am I missing?

I meant "rpm" as in Fedora/CentOS/Alma/RHEL/Rocky

AkihiroSuda commented 6 months ago

Can we close this?

Now we have:

vsoch commented 6 months ago

Yes! I have two variants (that are a bit simpler) that I'm using too. Thanks to you both for the help!