Node memory swap support

[ehashman]

Enhancement Description

• One-line enhancement description (can be used as a release note): Kubernetes nodes support swap memory.
• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/tree/master/keps/sig-node/2400-node-swap
• Discussion Link: https://docs.google.com/document/d/1CZtRtC8W8FwW_VWQLKP9DW_2H-hcLBcH9KBbukie67M/edit# Kubelet/Kubernetes should work with Swap Enabled kubernetes#53533
• Primary contact (assignee): @iholder101 @kannon92 @pacoxu
• Responsible SIGs: Node
• Enhancement target (which target equals to which milestone):
  • Alpha release target (x.y): 1.22
  • Beta1 release target (x.y): 1.28
  • Beta2 release target (x.y): 1.29-1.31
  • Stable release target (x.y): 1.32
• [x] Alpha
  • [x] KEP (k/enhancements) update PR(s): https://github.com/kubernetes/enhancements/pull/2602
  • [x] Code (k/k) update PR(s):
  • [x] CI/test infra changes: https://github.com/kubernetes/test-infra/pull/22602 https://github.com/kubernetes/test-infra/pull/22809
  • [x] CRI and kubelet changes https://github.com/kubernetes/kubernetes/pull/102823
  • [x] Docs (k/website) update PR(s): https://github.com/kubernetes/website/pull/28838
• [x] Beta1
  • [x] KEP (k/enhancements) update PR(s):
  • [x] https://github.com/kubernetes/enhancements/pull/2858
  • [x] https://github.com/kubernetes/enhancements/pull/3957
  • [x] Issues and Code (k/k) update PR(s):
  • Support cgroups v2 with auto-calculated LimitedSwap for Burstable QoS pods
    • [x] https://github.com/kubernetes/kubernetes/issues/119427
    • [x] https://github.com/kubernetes/kubernetes/issues/119428
    • [x] https://github.com/kubernetes/kubernetes/pull/118764
  • [x] Add Swap stats to /stats/summary and /metrics/resource
    • [x] https://github.com/kubernetes/kubernetes/issues/119425
    • [x] https://github.com/kubernetes/kubernetes/pull/118865
  • [x] Determine a set of metrics for measuring node performance with swap enabled
    • [x] https://github.com/kubernetes/kubernetes/issues/119424
    • [x] https://github.com/kubernetes/kubernetes/pull/118865
    • [x] https://github.com/google/cadvisor/pull/3339
  • [x] Improve test coverage and make sure tests are green
    • [x] https://github.com/kubernetes/kubernetes/issues/119430
    • [x] https://github.com/kubernetes/kubernetes/pull/118764
  • [x] Docs (k/website) update(s):
  • [x] https://github.com/kubernetes/website/pull/42009
  • [X] https://github.com/kubernetes/website/pull/42167
  • [x] Misc
  • [x] https://github.com/google/cadvisor/pull/3293
  • [x] https://github.com/kubernetes/kubernetes/pull/119225
  • Archive / Outdated
• [x] Beta2
  • [x] KEP (k/enhancements) update PR(s):
  • [x] 1.29 https://github.com/kubernetes/enhancements/pull/4275
  • [x] 1.30 https://github.com/kubernetes/enhancements/pull/4401
  • [x] 1.30 https://github.com/kubernetes/enhancements/pull/4484
  • [x] Issues and Code (k/k) update PR(s):
  • https://github.com/kubernetes/kubernetes/pull/122175
  • https://github.com/kubernetes/kubernetes/pull/122241
  • [x] 1.30
    • [x] https://github.com/kubernetes/kubernetes/pull/122745
    • [x] https://github.com/kubernetes/kubernetes/pull/123738
  • [x] 1.31
    • [x] https://github.com/kubernetes/kubernetes/pull/123557
    • [x] https://github.com/kubernetes/kubernetes/pull/123749
    • [x] 1.31 https://github.com/kubernetes/kubernetes/pull/124060
    • [x] 1.31 https://github.com/kubernetes/kubernetes/pull/125277
    • [x] 1.31 https://github.com/kubernetes/kubernetes/pull/125402
  • [x] Docs (k/website) update(s):
  • [x] 1.30 https://github.com/kubernetes/website/pull/45128
• [ ] GA (related PRs, can be assigned to a specific track later)
  • [ ] KEP (k/enhancements) update PR(s):
  • [ ] https://github.com/kubernetes/enhancements/pull/4701
  • [ ] Issues and Code (k/k) update PR(s):
  • [x] https://github.com/kubernetes/kubernetes/issues/120801
  • [ ] https://github.com/kubernetes/kubernetes/issues/120800
  • [x] https://github.com/kubernetes/kubernetes/issues/120798.
  • [x] https://github.com/kubernetes/kubernetes/issues/105978
  • [ ] https://github.com/kubernetes/kubernetes/issues/121820
  • Add the ability to set a system-reserved quantity of swap on a node
    • [ ] https://github.com/kubernetes/kubernetes/issues/120802
    • [x] https://github.com/kubernetes/kubernetes/issues/105019 dup by 120802
    • https://github.com/kubernetes/kubernetes/pull/105271
  • https://github.com/kubernetes/kubernetes/pull/119327
  • [x] https://github.com/kubernetes/kubernetes/issues/122279
  • [ ] Docs (k/website) update(s):

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

/sig node

[karan]

@cookieisaac

[ehashman]

/stage alpha /milestone v1.22

[jrsapi]

Greetings @ehashman! Enhancement shadow checking in with a few reminders. 1.22 Enhancements Freeze starts at 23:59:59 pst on Thursday, May 13. A few items needing review for this KEP:

• Enhancement README

• PRR approved

• https://github.com/kubernetes/enhancements/issues/2400 updated with current target and alpha information.

Thanks!

[ehashman]

All of this is covered in https://github.com/kubernetes/enhancements/pull/2602

[jrsapi]

Greetings @ehashman! Thanks for the follow-up. After reviewing the KEP and PRR everything looks on target. The enhancement is marked at risk but once merged we can move its status to tracked. A reminder that tomorrow 5/13 the Enhancement freeze starts at 23:59:59 PST.

Thanks!

[ehashman]

Work breakdown for 1.22

• Documentation for enabling swap (TBD - @ehashman?)
• CI environment/test updates (@ike-ma)
  • Build images with swap for 2 Linux distros
  • Add jobs to test-infra that use the images and enable the swap feature flag/kubelet option for node e2e suite
  • See also Test Plan
• CRI and kubelet changes (@ehashman)
  • Complete API changes per KEP
  • Add e2e tests as appropriate

External to k8s but still need to happen:

• Containerd update to use new CRI
• CRI-O update to use new CRI

Once the above CRI updates happen, ensure that CI environment is using latest container runtimes with updated CRI.

[jrsapi]

Greetings @ehashman , Enhancement shadow checking with a reminder that we are 2 weeks away from code freeze (July 8, 2021). Can you like the k/k PR/s that are needed to implement this enhancement for the 1.22 milestone?

[jrsapi]

Greetings @ehashman, A friendly reminder that code freeze is this Thursday, July 8th and we're tracking the following k/k PR:

• https://github.com/kubernetes/kubernetes/pull/102823 The PR has to be merged by code freeze and currently, this KEP is marked "At Risk". Also, Docs PR placeholder deadline is Friday, July 9th. Here are instructions on opening a PR in dev-1.22 branch.

Thanks!

[ehashman]

PR just merged. Docs placeholder is https://github.com/kubernetes/website/pull/28838

[jrsapi]

Awesome! Thanks for the update. Moving this to "Tracked".

[ehashman]

We are good to go for 1.22! Docs complete.

[ehashman]

/milestone v1.23

[ehashman]

/stage beta

[kevindelgado]

Hi @ehashman 1.23 Enhancements team here. Just checking in as we approach enhancements freeze on Thursday 09/09. Here's where this enhancement currently stands:

• [X] KEP file using the latest template has been merged into the k/enhancements repo.
• [X] KEP status is marked as implementable
• [X] KEP has a test plan section filled out.
• [X] KEP has up to date graduation criteria. ~(Graduation Criteria for beta currently says "tentative")~
• [X] KEP has a production readiness review that has been completed and merged into k/enhancements.

~Looks like for this one we would just need the KEP/PRR to merge (and the "tentative" label on graduation criteria removed) by enhancements freeze :)~

Good to go now!

Thanks!

[kevindelgado]

Hi @ehashman

Ping! As a reminder your PR (#2858) needs to merge by EOD PST tomorrow September 9th to be included in the 1.23 Release. After that time you will need to request an exception.

Lmk if you need anything, Kevin

[ehashman]

Hello @kevindelgado,

PRR is now approved and the "tentative" bit has been moved to GA. Awaiting final node approval.

[jlbutler]

Hi @ehasman :wave: 1.23 Docs lead here.

This enhancement is marked as 'Needs Docs' for the 1.23 release.

Please open a PR against the dev-1.23 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thu November 18, 11:59 PM PDT.

Thanks!

[SergeyKanzhelev]

As discussed at SIG Node meeting on October 26, 2021 we are removing this enhancement from 1.23 milestone. @kubernetes/release-managers can you please remove the milestone label?

[endocrimes]

Swap support has now landed in containerd: https://github.com/containerd/containerd/pull/6320 - Have a branch prepped for when a release drops to include in test-infra.

[ehashman]

/milestone v1.24

[yangjunmyfm192085]

/cc

[gracenng]

Hi @ehashman ! 1.24 Enhancements team here. Just checking in as we approach enhancements freeze on 18:00pm PT on Thursday Feb 3rd. This enhancements is targeting beta for 1.24.

The status of this enhancement is track as tracked , no action is required for Enhancements Freeze. Thanks!

[nate-double-u]

Hi @ehashman :wave: 1.24 Docs lead here.

This enhancement is marked as Needs Docs for the 1.24 release.

Please follow the steps detailed in the documentation to open a PR against the dev-1.24 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, March 31st, 2022 @ 18:00 PDT.

Also, if needed take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thanks!

[ehashman]

Work for the 1.24 release was organized on a project board: https://github.com/orgs/kubernetes/projects/69

We also have a slack channel: #sig-node-swap

I am doubtful this will make the soft node freeze on Mar. 4 (this Friday) given how much work is left to complete.

[ehashman]

/milestone clear per https://docs.google.com/document/d/1Ne57gvidMEWXR70OxxnRkYquAoMpt56o75oZtg-OeBg/edit#bookmark=id.w24ewvhebv4u

[gracenng]

Removing this enhancements from the release

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[pacoxu]

/remove-lifecycle stale

[marosset]

/milestone v1.26 /label lead-opted-in (I'm doing this on behalf of @ruiwen-zhao / SIG-node)

[rhockenbury]

Hello @ehashman 👋, 1.26 Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT on Thursday 6th October 2022.

This enhancement is targeting for stage alpha for 1.26 (from this comment, we'll track it as a major change for alpha, rather than beta graduation)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: 1.26
• [ ] KEP readme has a updated detailed test plan section filled out
• [ ] KEP readme has up to date graduation criteria
• [X] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would just need to update the following:

• Update latest-milestone, stage and milestone in the KEP.yaml
• Update the test plan section to match the latest template
• Merge up https://github.com/kubernetes/enhancements/pull/3571

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[rhockenbury]

/stage alpha /label tracked/yes

[derekwaynecarr]

/remove-label lead-opted-in

no volunteer has come forward yet this release to push this work further, it will remain unchanged in 1.26.

[SergeyKanzhelev]

/remove-label lead-opted-in

no volunteer has come forward yet this release to push this work further, it will remain unchanged in 1.26.

@derekwaynecarr I plan to move this forward, not to beta, but to more clarity. See https://github.com/kubernetes/enhancements/pull/3571.

[rhockenbury]

/label lead-opted-in

[rhockenbury]

It looks like we would still need to update the KEP readme to use the latest template to meet all requirements for enhancements freeze.

[rhockenbury]

Hello 👋, 1.26 Enhancements Lead here.

Unfortunately, this enhancement did not meet requirements for enhancements freeze.

If you still wish to progress this enhancement in v1.26, please file an exception request. Thanks!

/milestone clear /label tracked/no /remove-label tracked/yes /remove-label lead-opted-in

[SergeyKanzhelev]

/assign

@rhockenbury I changed the formatting of TestPlan and added some links to tests we introduced in Alpha stage. Can this be returned back to milestone without the exception process since the content was not modified?

[SergeyKanzhelev]

/unassign @ehashman

(@ehashman please feel free to re-assign back to yourself)

[rhockenbury]

Alright, thanks for making those changes. I have it marked as tracked now but can you please also update the KEP yaml to set stage: alpha? - https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/2400-node-swap/kep.yaml#L21

/milestone v1.26 /label tracked/yes /label lead-opted-in /remove-label tracked/no

[ehashman]

@SergeyKanzhelev I am not planning on working on this at this time, please feel free to continue driving it forward and let me know if you need any feedback/reviews.

[Atharva-Shinde]

Hey @ehashman @SergeyKanzhelev 👋,

Checking in as we approach 1.26 code freeze at 17:00 PDT on Tuesday 8th November 2022.

Please ensure the following items are completed:

• [ ] All PRs to the Kubernetes repo that are related to your enhancement are linked in the issue description above(for tracking purposes). If there are any other PRs beyond:
  https://github.com/kubernetes/kubernetes/issues/104292 https://github.com/kubernetes/kubernetes/issues/105019 https://github.com/kubernetes/kubernetes/issues/105020 https://github.com/kubernetes/kubernetes/issues/105021 https://github.com/kubernetes/kubernetes/issues/105022 https://github.com/kubernetes/kubernetes/issues/105023 https://github.com/kubernetes/kubernetes/issues/105025 https://github.com/kubernetes/kubernetes/issues/105978 please include them in the issue description too.
• [ ] All PRs are fully merged by the code freeze deadline.

As always, we are here to help should questions come up. Thanks :)

[cathchu]

Hello @SergeyKanzhelev 👋 1.26 Release Docs shadow here!

This enhancement is marked as ‘Needs Docs’ for 1.26 release. Please follow the steps detailed in the documentation to open a PR against dev-1.26 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by November 9. Also, take a look at Documenting for a release to familiarize yourself with the docs requirement for the release.

Thank you!

[Atharva-Shinde]

Hey @ehashman @SergeyKanzhelev 👋, just a quick check-in again before 1.26 code freeze at 17:00 PDT Tuesday 8th November 2022 i.e tomorrow. Looks like we would at least need to get the code PR/s: https://github.com/kubernetes/kubernetes/issues/105019 https://github.com/kubernetes/kubernetes/issues/105020 https://github.com/kubernetes/kubernetes/issues/105021 https://github.com/kubernetes/kubernetes/issues/105022 https://github.com/kubernetes/kubernetes/issues/105023 https://github.com/kubernetes/kubernetes/issues/105025 https://github.com/kubernetes/kubernetes/issues/105978 (any other PRs?) merged before the code-freeze :)

[rhockenbury]

Hello 👋, 1.26 Enhancements Lead here.

Unfortunately, this enhancement did not meet requirements for code freeze.

If you still wish to progress this enhancement in v1.26, please file an exception request. Thanks!

/milestone clear /label tracked/no /remove-label tracked/yes /remove-label lead-opted-in

[fabiand]

Hey. We've valiated that swap works nice with KubeVirt workloads. What kind of data is needed in order to lift this feature to beta?

[drvnoo]

I think this is the most important missing feature in K8s currently. Thanks everyone for all the amazing work being done here. This is a game changer!

[dchen1107]

/label lead-opted-in /milestone v1.27

[k8s-ci-robot]

@dchen1107: Can not set label lead-opted-in: Must be member in one of these teams: [release-team-enhancements release-team-leads sig-api-machinery-leads sig-apps-leads sig-architecture-leads sig-auth-leads sig-autoscaling-leads sig-cli-leads sig-cloud-provider-leads sig-cluster-lifecycle-leads sig-contributor-experience-leads sig-docs-leads sig-instrumentation-leads sig-k8s-infra-leads sig-multicluster-leads sig-network-leads sig-node-leads sig-release-leads sig-scalability-leads sig-scheduling-leads sig-security-leads sig-storage-leads sig-testing-leads sig-windows-leads]

In response to [this](https://github.com/kubernetes/enhancements/issues/2400#issuecomment-1410962086): >/label lead-opted-in >/milestone v1.27 > Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[logicalhan]

/label lead-opted-in

[SergeyKanzhelev]

@marosset this KEP is not moving to the different stage, but we will be working on it for "Alpha2" milestone. Is there anything needed for this KEP to be tracked for 1.27?