Update the kernel configuration for cgroup v2

KentaTada commented 1 year ago

Kubernetes 1.25 brings cgroup v2 to GA. cgroup v2 needs some additional kernel configs. For example, you need to enable CONFIG_CGROUP_BPF if you want to use the the device controller. When it comes to Kubernetes, I have never investigated what config is actually needed. But I just create an issue at first.

pacoxu commented 1 year ago

https://github.com/kubernetes/system-validators/pull/12#issuecomment-607668321 by @odinuge

CONFIG_CGROUP_BPF - Required for cgroupv2 (for controlling devices)

https://github.com/containerd/containerd/pull/3799#issuecomment-555740694 by @AkihiroSuda

kernel >= 4.15 with CONFIG_CGROUP_DEVICE and CONFIG_CGROUP_BPF is required.

/cc @bobbypage @mrunalp for cgroup v2 GA

https://github.com/kubernetes/minikube/pull/6572

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 7 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle rotten
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 6 months ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Reopen this issue with /reopen
Mark this issue as fresh with /remove-lifecycle rotten
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 6 months ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/system-validators/issues/36#issuecomment-1950933640): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

neolit123 commented 6 months ago

/remove-lifecycle rotten @pacoxu @KentaTada do we still need such a change?

KentaTada commented 6 months ago

Yes. We need to investigate and prepare for the kernel configuration for cgroupv2. If I have time, I'll investigate it.

neolit123 commented 6 months ago

thanks, @KentaTada

k8s-triage-robot commented 3 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

After 90d of inactivity, lifecycle/stale is applied
After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

Mark this issue as fresh with /remove-lifecycle stale
Close this issue with /close
Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

KentaTada commented 3 months ago

/remove-lifecycle stale

pacoxu commented 1 month ago

The v1.31 KEP https://github.com/kubernetes/enhancements/issues/4569

https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4569-cgroup-v1-maintenance-mode/README.md KEP-4569: Moving cgroup v1 support into maintenance mode

I opened https://github.com/kubernetes/system-validators/pull/37 for the kernel version.

for KernelConfig part, I need to check the list.

neolit123 commented 1 month ago

Yes. We need to investigate and prepare for the kernel configuration for cgroupv2. If I have time, I'll investigate it.

@KentaTada should we include this in the next system-validators release too?

@pacoxu recently added:

KentaTada commented 1 month ago

@KentaTada should we include this in the next system-validators release too?

Yes. Although I haven't completely caught up with these commits yet, we should include the info about cgroup v2 before moving cgroup v1 support into maintenance mode.

BTW, we also need to update the list of kernel config. For example, CONFIG_CGROUP_FREEZER is not actually required for cgroup v2. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/init/Kconfig?h=v5.8#n1006 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/init/Kconfig?h=v6.10#n1103

neolit123 commented 1 month ago

would you have time to send a PR or give us the info of all the required changes that we need to do?

pacoxu commented 1 month ago

we also need to update the list of kernel config. For example, CONFIG_CGROUP_FREEZER is not actually required for cgroup v2. https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/init/Kconfig?h=v5.8#n1006 https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/init/Kconfig?h=v6.10#n1103

For cgroup v2, we use cgroup.freeze instead, which needs kernel 5.2+.

KentaTada commented 1 month ago

I'm sorry but I need to support my family this week. In addition to that, I need to chair a meeting of KubeDay and my local CNCF-related eBPF event in August. https://sched.co/1eh9w https://community.cncf.io/e/m4e8cg/

For cgroup v2, we use cgroup.freeze instead, which needs kernel 5.2+.

What I wanted to say is that we need to prepare for how to detect what cgroup v2 features are available. Unlike in v1, the kernel config alone cannot determine whether the required v2 features are enabled. Maybe, it is possible if this validator checks for the existence of the file of cgroup.freeze. We also need to think about BPF-based interfaces like the device controller. In addition to that, we need to confirm what v2 features are currently needed for k8s at first. This change should be made with caution because k8s users all over the world may recompile their kernel by the result of kubeadm.

neolit123 commented 1 month ago

I'm sorry but I need to support my family this week. In addition to that, I need to chair a meeting of KubeDay and my local CNCF-related eBPF event in August.

1.31 was just released so this is planned for 1.32. we have a whole k8s release cycle to tackle the feature detection for cgroups v2.

What I wanted to say is that we need to prepare for how to detect what cgroup v2 features are available. Unlike in v1, the kernel config alone cannot determine whether the required v2 features are enabled. Maybe, it is possible if this validator checks for the existence of the file of cgroup.freeze. We also need to think about BPF-based interfaces like the device controller.

i think having the cgroups v2 validation load additional files should be fine.

In addition to that, we need to confirm what v2 features are currently needed for k8s at first.

@pacoxu maybe we need to ask SIG node and bring them this this thread?

This change should be made with caution because k8s users all over the world may recompile their kernel by the result of kubeadm.

completely agree.

pacoxu commented 1 month ago

What I wanted to say is that we need to prepare for how to detect what cgroup v2 features are available. Unlike in v1, the kernel config alone cannot determine whether the required v2 features are enabled. Maybe, it is possible if this validator checks for the existence of the file of cgroup.freeze. We also need to think about BPF-based interfaces like the device controller.

i think having the cgroups v2 validation load additional files should be fine.

In https://github.com/kubernetes/kubernetes/pull/126595, we may use cpu.stat check instead of version check there.

kubernetes / system-validators

Update the kernel configuration for cgroup v2 #36