search

kubernetes / minikube

Run Kubernetes locally
https://minikube.sigs.k8s.io/
Apache License 2.0
29.17k stars 4.87k forks source link

ISO upgrades to support user namespaces #18488

Open mbaynton opened 5 months ago

mbaynton commented 5 months ago

What Happened?

A new feature is nearing readiness in kubernetes/containerd/runc that enables you to run your containers in their own linux user namespace. This is beneficial for security / container isolation and also enables you to do certain operations inside containers that were previously only possible with privileged host permissions.

This feature is set to be promoted to beta in kubernetes 1.30.

It would be great to be able to use minikube to prototype setups that use user namespaces, but lots of much newer software in the ISO will be needed:

Attach the log file

n/a, trying to submit a feature request but can't find a better form

Operating System

Ubuntu

Driver

KVM2

afbjorklund commented 5 months ago

Sounds like an experimental feature, that doesn't really fit with the LTS versions used in the minikube ISO (or KIC)

Theoretically there could be two image versions, but that means more maintenance (and it is already struggling)

afbjorklund commented 5 months ago

KEP 127 looks completely untested/unimplemented for Docker, which is still the minikube default container runtime...

For Beta, the feature is tested for containerd and CRI-O in cri-tools repo using critest

rata commented 5 months ago

@afbjorklund if it helps to make a decision, it doesn't seem it will be implemented in cri-dockerd anytime soon, as docker has some limitations. I've been trying them to implement it for years now, and although they like the idea, there was no progress so far: https://github.com/Mirantis/cri-dockerd/issues/74

mbaynton commented 5 months ago

doesn't really fit with the LTS versions used in the minikube ISO

Yeah at the moment it's definitely too soon to build an updated ISO for general use that enables this. I'm optimistic releases of containerd and runc will be available soon though :). But everything needs to be marked as an LTS release as well?

afbjorklund commented 5 months ago

But everything needs to be marked as an LTS release as well?

It is not an absolute rule, but historically minikube used Ubuntu LTS as the base for decisions for the Buildroot versions. Other distributions such as kind, were doing latest/greatest (or Debian) and rebuilding containerd on a nightly basis...

Ubuntu Linux
16.04 4.4
18.04 4.15
20.04 5.4
22.04 5.15
24.04 6.8 (?)

But after that it was pretty arbitrary, stayed on 4.19 for a while since it was stable and then jumped to 5.10 (770d41f21b42b037fbf4386b1817e3a3d3370aaf). One could go for a similar jump to 6.x, perhaps timed with the OS update. The runtime versions are even more arbitrary.

The versions of containerd and runc usually came (bundled) with Docker...

https://docs.docker.com/engine/install/binaries/ (or get.docker.com)

afbjorklund commented 5 months ago

Note: Buildroot 2024.02 (LTS) supports these kernels:

# From https://www.kernel.org/pub/linux/kernel/v6.x/sha256sums.asc
sha256  4e43d8c5fba14f7c82597838011648056487b7550fd83276ad534559e8499b1d  linux-6.6.18.tar.xz
sha256  faa49ca22fb55ed4d5ca2a55e07dd10e4e171cfc3b92568a631453cd2068b39b  linux-6.1.79.tar.xz
# From https://www.kernel.org/pub/linux/kernel/v5.x/sha256sums.asc
sha256  bd84809a367eb400eb04e0e70294e6ba12fc03b6bfb5a7dfaca548f8947501b0  linux-5.15.149.tar.xz
sha256  4ea63c5a90fdc3c459ab35c11ee8c93d2364a7cdbfb101100f8cab70d490ef6d  linux-5.10.210.tar.xz
sha256  ff54bec6d053c7994f3bb8c45021de2858ff9f740d2ccbbcf072b87821a918cf  linux-5.4.269.tar.xz
# From https://www.kernel.org/pub/linux/kernel/v4.x/sha256sums.asc
sha256  83eeff613405d0045d0f717c6ac14c178678fe0a163c41d9dd8878ac0f73e352  linux-4.19.307.tar.xz

See https://buildroot.org/news.html (and 2024.02.x)

Going with Kernel 5.15 would be the more obvious choice.

mbaynton commented 5 months ago

The user namespace stuff needs features from newer kernels than 5.15. I've had success with 6.6, apparently you can go back as far as 6.3 but it looks like that's not an option for buildroot.

rata commented 5 months ago

@mbaynton it should work fine with 5.12+ but you can't use tmpfs volumes (so service account tokens can't really work) and you have a higher pod start latency and storage overhead if you use 5.12 too.

k8s-triage-robot commented 2 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

rata commented 1 month ago

/remove-lifecycle rotten

containerd 2.0 and runc 1.2 should be released this year with proper support. It seems worth keeping this open a little longer

medyagh commented 2 weeks ago

I agree we should have a tutorial that lets ppl build their own ISO easily and basicly if they want containerd 2.0 can do minkube start --iso-url=file://....

and not part of released minikube ISOs