Fine-grained SupplementalGroups control

[everpeace]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add new API surface to control and track how supplemental groups are applied in the container.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/3619-supplemental-groups-policy
• Discussion Link: https://github.com/kubernetes/kubernetes/issues/112879
• Primary contact (assignee):　@everpeace
• Responsible SIGs: sig-node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.31
  • Beta release target (x.y): ~1.32~ --> 1.33 or later (depends on Containerd v2 release and it's adoption)
  • Stable release target (x.y):
• [x] Alpha
  • [x] KEP (k/enhancements) update PR(s):
  • [x] https://github.com/kubernetes/enhancements/pull/3620
  • [x] https://github.com/kubernetes/enhancements/pull/3862
  • [x] https://github.com/kubernetes/enhancements/pull/3874
  • [x] https://github.com/kubernetes/enhancements/pull/4628
  • [x] https://github.com/kubernetes/enhancements/pull/4728
  • [x] Code (k/k) update PR(s):
  • [x] https://github.com/kubernetes/kubernetes/pull/117842
  • [x] https://github.com/kubernetes/kubernetes/pull/125470
  • [x] Docs (k/website) update PR(s):
  • [x] https://github.com/kubernetes/website/pull/46920
  • [x] https://github.com/kubernetes/website/pull/46921
  • contained (optional):
  • [x] https://github.com/containerd/containerd/pull/9737
  • [x] https://github.com/containerd/containerd/pull/10410
  • CRI-O (optional):
  • [x] https://github.com/cri-o/cri-o/pull/8268
  • [x] https://github.com/cri-o/cri-o/pull/8386
  • cri-tools (optional):
  • [x] https://github.com/kubernetes-sigs/cri-tools/pull/1438
• [ ] Beta
  • [ ] KEP (k/enhancements) update PR(s):
  • [x] https://github.com/kubernetes/enhancements/pull/4809
  • [ ] https://github.com/kubernetes/enhancements/pull/4895
  • [ ] Code (k/k) update PR(s):
  • [ ] Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/assign /sig node /kind feature

[everpeace]

/assign /sig scheduling

[k8s-ci-robot]

@everpeace: The label(s) /remove-label sig/scheduling cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, lead-opted-in, tracked/no, tracked/out-of-tree, tracked/yes. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to [this](https://github.com/kubernetes/enhancements/issues/3619#issuecomment-1278480589): >/remove-label sig/scheduling Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[everpeace]

/kind feature

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[everpeace]

/remove-lifecycle stale

[SergeyKanzhelev]

/milestone v1.27

let's see if we can start it in 1.27

[SergeyKanzhelev]

/label lead-opted-in

[SergeyKanzhelev]

/stage alpha

[npolshakova]

Hello @everpeace 👋, 1.27 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for 1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: 1.27
• [ ] KEP readme has a updated detailed test plan section filled out
• [ ] KEP readme has up to date graduation criteria
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements.

It looks like https://github.com/kubernetes/enhancements/pull/3620 will address most of these issues.

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[SergeyKanzhelev]

@npolshakova, kep yaml has correct values for stage, milestone, etc:

https://github.com/kubernetes/enhancements/blob/472a381fb32767bf65cbd28b62c04ee655cc2408/keps/sig-node/3619-supplemental-groups-policy/kep.yaml#L8-L30

This KEP updates to the latest template and covers testing section: https://github.com/kubernetes/enhancements/pull/3862

PRR review completed:

https://github.com/kubernetes/enhancements/blob/472a381fb32767bf65cbd28b62c04ee655cc2408/keps/prod-readiness/sig-node/3619.yaml#L1-L6

So once #3862 is merged, this KEP is ready for the milestone

[npolshakova]

Great! Looks like https://github.com/kubernetes/enhancements/pull/3862 went in so this enhancement as tracked for v1.27. Thanks!

/label tracked/yes

[npolshakova]

Hi @everpeace,

Checking in as we approach 1.27 code freeze at 17:00 PDT on Tuesday 14th March 2023.

Please ensure the following items are completed:

• [ ] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• [ ] All PRs are fully merged by the code freeze deadline.

Please let me know if there are any other PRs in k/k I should be tracking for this KEP. As always, we are here to help should questions come up. Thanks!

[taniaduggal]

Hi @everpeace 👋 , I’m reaching out from the 1.27 Release Docs team. This enhancement is marked as ‘Needs Docs’ for the 1.27 release. Please follow the steps detailed in the documentation to open a PR against dev-1.27 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by March 16. For more information, please take a look at Documenting for a release to familiarize yourself with the documentation requirements for the release. Please feel free to reach out with any questions. Thanks!

[npolshakova]

Hi @everpeace, this is the status as we approach code freeze today:

• [ ] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• [ ] All PRs are fully merged by the code freeze deadline.

Please let me know what other PRs in k/k I should be tracking for this KEP.

As always, we are here to help should questions come up. Thanks!

[npolshakova]

Unfortunately the implementation PRs associated with this enhancement have not merged by code-freeze so this enhancement is getting removed from the release.

If you would like to file an exception please see https://github.com/kubernetes/sig-release/blob/master/releases/EXCEPTIONS.md

/milestone clear /remove-label tracked/yes /label tracked/no

[SergeyKanzhelev]

@everpeace do you plan to continue working on this KEP in 1.28?

[everpeace]

@SergeyKanzhelev Yes, I will continue working on this KEP. As I wrote in https://github.com/kubernetes/enhancements/pull/3620#issuecomment-1500728236 , my cleanup PR in containerd took much time to merge than I expected.

[everpeace]

Memo: Tasklist to alpha(v1.28)

• [ ] implement SupplementalGroupsPolicy in k/k (incl. updating cri-api)
• [ ] implement cri-api's SupplementalGroupsPolicy in containerd
• [ ] release containerd with cri-api's SupplementalGroupsPolicy support
• [ ] implement e2e for SupplementalGroupsPolicy
• [ ] add SupplementalGroupsPolicy to cri-test
• [ ] add SupplementalGroupsPolicy to k/website (https://kubernetes.io/docs/tasks/configure-pod-container/security-context/)

[SergeyKanzhelev]

/milestone v1.28

[thockin]

Is this still hoping to land alpha in 1.28?

[SergeyKanzhelev]

/label lead-opted-in

[npolshakova]

Hello @everpeace 👋, 1.28 Enhancements team here!

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for 1.28 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: 1.28
• [x] KEP readme has a updated detailed test plan section filled out
• [x] KEP readme has up to date graduation criteria
• [x] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would just need to update the following:

• Update latest milestone to 1.28
• Update alpha milestone to 1.28

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[SergeyKanzhelev]

For this KEP, we would just need to update the following:

• Update latest milestone to 1.28
• Update alpha milestone to 1.28

this is done, KEP should be ok to be tracked

[npolshakova]

With all the requirements fulfilled this enhancement is marked as tracked for the Enhancements freeze 🚀

[taniaduggal]

Hello @everpeace 1.28 Docs Shadow here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[npolshakova]

Hey again @everpeace :wave:

Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

Here’s the enhancement’s state for the upcoming code freeze:

• [ ] All the PRs that are related to your enhancement are linked in the above issue description (for tracking purposes). This includes code, tests, and documentation related PR/s.
• [ ] All code related PR/s are merged or are in merge-ready state ( i.e they have approved and lgtm labels applied) by the code freeze deadline. This includes any tests related PR/s too.

Also please let me know if there are other PRs in k/k we should be tracking for this KEP. As always, we are here to help if any questions come up. Thanks!

[Rishit-dagli]

Hey @everpeace , could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.

[taniaduggal]

Hey @everpeace, Docs Shadow here! could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.

[Atharva-Shinde]

Hello @everpeace 👋, 1.28 Enhancements Lead here.

Unfortunately, the implementation (code related) PR associated with this enhancement was not in the merge-ready state by the code-freeze and hence this enhancement is now being removed from the v1.28 milestone.

If you still wish to progress this enhancement in v1.28, please file an exception request. Thanks!

/milestone clear

[SergeyKanzhelev]

@everpeace do you plan to keep working on this KEP for 1.29?

[SergeyKanzhelev]

/remove-label lead-opted-in

while we are confirming the owner for the KEP

[everpeace]

@SergeyKanzhelev Sorry for my long absence. Yes, let me work on this for 1.29.

[npolshakova]

Hello @everpeace, 1.29 Enhancements team here! Is this enhancement targeting 1.29? If it is, can you follow the instructions here to opt in the enhancement and make sure the lead-opted-in label is set so it can get added to the tracking board? Thanks!

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[everpeace]

/remove-lifecycle stale

[everpeace]

@kubernetes/sig-node-leads Recently, I have been working (rebooted) on this KEP actively to target v1.31.

• k/k: https://github.com/kubernetes/kubernetes/pull/117842
• containerd: https://github.com/containerd/containerd/pull/9737
  • it already got LGTM except for cri-api dependency.

Could we get this KEP opt-in for v1.31? WDYT?

[thockin]

@mrunalp or @dchen1107 or delegate - are you OK to lead opt-in ?

[mrunalp]

/label lead-opted-in

[ArkaSaha30]

Hello @everpeace 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 02:00 UTC Friday 14th June 2024 / 19:00 PDT Thursday 13th June 2024.

This enhancement is targeting stage alpha for v1.31 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [x] KEP readme using the latest template has been merged into the k/enhancements repo.
• [x] KEP status is marked as implementable for latest-milestone: v1.31. KEPs targeting stable will need to be marked as implemented after code PRs are merged and the feature gates are removed.
• [x] KEP readme has up-to-date graduation criteria
• [x] KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here). If your production readiness review is not completed yet, please make sure to fill the production readiness questionnaire in your KEP by the PRR Freeze deadline so that the PRR team has enough time to review your KEP before the enhancements freeze.

All the KEP requirements are in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀

The status of this enhancement is marked as tracked for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[LaurentGoderre]

@everpeace Docs team here. Can you create a PR for the documentation for this enhancement?

[rashansmith]

Hi @everpeace,

:wave: from the v1.31 Communications Team! We'd love for you to opt in to write a feature blog about your enhancement! Some reasons why you might want to write a blog for this feature include (but are not limited to) if this introduces breaking changes, is important to our users, or has been in progress for a long time and is graduating.

To opt in, let us know and open a Feature Blog placeholder PR against the website repository by 3rd July, 2024. For more information about writing a blog see the blog contribution guidelines.

Note: In your placeholder PR, use XX characters for the blog date in the front matter and file name. We will work with you on updating the PR with the publication date once we have a final number of feature blogs for this release.

[everpeace]

@LaurentGoderre

Docs team here. Can you create a PR for the documentation for this enhancement?

Thanks, I opened a draft PR for this KEP: https://github.com/kubernetes/website/pull/46920

[everpeace]

@rashansmith

To opt in, let us know and open a Feature Blog placeholder PR against the website repository by 3rd July, 2024. For more information about writing a blog see the blog contribution guidelines.

Thanks, I'm happy to write a blog for this KEP. I opened a placeholder PR: https://github.com/kubernetes/website/pull/46921

[ArkaSaha30]

Hey again @everpeace 👋, Enhancements team here,

Just checking in as we approach code freeze at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

Here's where this enhancement currently stands:

• [x] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• [ ] All PR/s are ready to be merged (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

Regarding this enhancement, it appears that there are currently 2 open PRs in the k/k repository out of which one is merged and the other is up for review.

For this KEP, we would need to do the following:

• Ensure the remaining PR is prepared for merging (they have approved and lgtm labels applied) by the code freeze deadline. This includes tests.

If you anticipate missing code freeze, you can file an exception request in advance.

The status of this enhancement is currently marked as at risk for code freeze.

[everpeace]

Just checking in as we approach code freeze at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024. Regarding this enhancement, it appears that there are currently 2 open PRs in the k/k repository out of which one is https://github.com/kubernetes/kubernetes/pull/117842(merged) and the other is up for https://github.com/kubernetes/kubernetes/pull/125470#pullrequestreview-2153057910(in review).

@thockin @mrunalp @SergeyKanzhelev (as KEP reviwers/approvers) cc/ @haircommander (as an active reviwer on this KEP PRs)

I apologize for bothering you again. As you may already be aware, my last k/k PR https://github.com/kubernetes/kubernetes/pull/125470#pullrequestreview-2153057910 for this KEP is ready for review. I would be very grateful if this KEP could reach alpha in v1.31🙇.

[ArkaSaha30]

Hey again @everpeace 👋, 1.31 Enhancements team here,

Just a quick friendly reminder as we approach code freeze in about 2 days, at 02:00 UTC Wednesday 24th July 2024 / 19:00 PDT Tuesday 23rd July 2024.

The current status of this enhancement is marked as at risk for code freeze. A few requirements mentioned in the comment https://github.com/kubernetes/enhancements/issues/3619#issuecomment-2215251778 still need to be completed. The following PR as per the description still needs to be merged:

• [ ] https://github.com/containerd/containerd/pull/10410

If you anticipate missing code freeze, you can file an exception request in advance.

[thockin]

Whether the containerd PR is merged or not shouldn't matter here - their release train is async to ours anyway. @ArkaSaha30 tell me if you disagree?

[ArkaSaha30]

Whether the containerd PR is merged or not shouldn't matter here - their release train is async to ours anyway.

Yes, agree. Thank you for the clarification. Since the code PRs to k/k are in place and merged, I am marking this enhancement as tracked for code freeze for the 1.31 Code Freeze!

[everpeace]

Thanks, I updated this Issue's description to make containerd/CRI-O PRs optional.

updated: all the optional PR are merged now 👍 The remaining part is just doc and blog.