Kubelet limit of Parallel Image Pulls

[ruiwen-zhao]

Enhancement Description

• One-line enhancement description (can be used as a release note): Add a node-level limit to kubelet to limit the number of parallel image pulls

• Kubernetes Enhancement Proposal: https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/3673-kubelet-parallel-image-pull-limit/README.md

• Discussion Link:

• Primary contact (assignee): @pacoxu @ruiwen-zhao

• Responsible SIGs: SIG Node

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.27
  • Beta release target (x.y): 1.32
  • Stable release target (x.y):

• [x] Alpha

  • [x] KEP (k/enhancements) update PR(s):
  • https://github.com/kubernetes/enhancements/pull/3713
  • [x] Code (k/k) update PR(s):
  • https://github.com/kubernetes/kubernetes/pull/115220
  • [x] Docs (k/website) update PR(s):
  • https://github.com/kubernetes/website/pull/39732

• [ ] Beta

  • [x] KEP (k/enhancements) update PR(s):
  • [x] 1.28 https://github.com/kubernetes/enhancements/pull/4036;
  • [x] 1.29 https://github.com/kubernetes/enhancements/pull/4173
  • [x] 1.30 https://github.com/kubernetes/enhancements/pull/4452
  • [x] Code (k/k) update PR(s):
  • [x] 1.32 https://github.com/kubernetes/kubernetes/pull/121604
  • [ ] Docs (k/website) update(s):
  • [x] https://github.com/kubernetes/website/pull/42061(mistakenly merged in v1.28, and reverted).

Related issues：

• [ ] https://github.com/kubernetes/kubernetes/issues/125164

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

[ruiwen-zhao]

/sig node

[pacoxu]

Some intial ideas can be found in https://docs.google.com/document/d/1MMZlKSxkDBDz0kEn9JGm848_IJgydJqXWz5t9D3Ev8I/edit# and https://github.com/kubernetes/kubernetes/pull/112242 https://github.com/kubernetes/kubernetes/pull/112044

[wojtek-t]

/lead opted-in /milestone v1.27

[wojtek-t]

/label lead-opted-in

[marosset]

Hello @pacoxu & @ruiwen-zhao 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for v1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: v1.27
• [ ] KEP readme has a updated detailed test plan section filled out
• [ ] KEP readme has up to date graduation criteria
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this enhancement, it looks like https://github.com/kubernetes/enhancements/pull/3713 will address most of these requirements. Please be sure to also:

• Update the status to implemnetable in the kep.yaml (when ready)

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[ruiwen-zhao]

Hello @pacoxu & @ruiwen-zhao 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on 18:00 PDT Thursday 9th February 2023.

This enhancement is targeting for stage alpha for v1.27 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable for latest-milestone: v1.27
• [ ] KEP readme has a updated detailed test plan section filled out
• [ ] KEP readme has up to date graduation criteria
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this enhancement, it looks like #3713 will address most of these requirements. Please be sure to also:

• Update the status to implemnetable in the kep.yaml (when ready)

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

Updated the KEP PR https://github.com/kubernetes/enhancements/pull/3713 to change the status to implemnetable. Once it is merged then all of the requirements will be addressed.

[marosset]

Thanks @ruiwen-zhao - this enhancement is ready to be tracked for v1.27

[pacoxu]

/assign @pacoxu @ruiwen-zhao

[ruiwen-zhao]

Draft document update: https://github.com/kubernetes/website/pull/39732

[mickeyboxell]

Hi @ruiwen-zhao @pacoxu 👋, I’m reaching out from the 1.27 Release Docs team. This enhancement is marked as ‘Needs Docs’ for the 1.27 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.27 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by March 16. For more information, please take a look at Documenting for a release to familiarize yourself with the documentation requirements for the release.

Please feel free to reach out with any questions. Thanks!

[ruiwen-zhao]

Hi @ruiwen-zhao @pacoxu 👋, I’m reaching out from the 1.27 Release Docs team. This enhancement is marked as ‘Needs Docs’ for the 1.27 release.

Please follow the steps detailed in the documentation to open a PR against dev-1.27 branch in the k/website repo. This PR can be just a placeholder at this time, and must be created by March 16. For more information, please take a look at Documenting for a release to familiarize yourself with the documentation requirements for the release.

Please feel free to reach out with any questions. Thanks!

Is https://github.com/kubernetes/website/pull/39732 sufficient for now?

[marosset]

Hi @ruiwen-zhao @pacoxu :wave:,

Checking in as we approach 1.27 code freeze at 17:00 PDT on Tuesday 14th March 2023.

Please ensure the following items are completed:

• [x] All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• [x] All PRs are fully merged by the code freeze deadline.

Please let me know if there are any other PRs in k/k I should be tracking for this KEP. I see the following PR was closed (https://github.com/kubernetes/kubernetes/pull/112242) - Is it going to be replaced with another PR?

As always, we are here to help should questions come up. Thanks!

[ruiwen-zhao]

Hi @marosset

The PR is https://github.com/kubernetes/kubernetes/pull/115220, which has been merged already. Thanks!

[SergeyKanzhelev]

this will be progressing to beta in 1.28 as per the sig node meeting 5/2/2023

/milestone v1.28

[pacoxu]

this will be progressing to beta in 1.28 as per the sig node meeting 5/2/2023

I did not attend the meeting. @ruiwen-zhao could you share some details about the plan of this feature in v1.28?

[ruiwen-zhao]

this will be progressing to beta in 1.28 as per the sig node meeting 5/2/2023

I did not attend the meeting. @ruiwen-zhao could you share some details about the plan of this feature in v1.28?

I am thinking of adding a e2e test for parallel image pulling with a MaxParallelImagePulls set. And that will be the only criteria for graduating to beta. Plus GKE starts using this feature in 1.27, so we could gather some feedback if this feature causes any regression.

WDYT?

[pacoxu]

If GKE will use it， the feedback would be very valuable.

To me， the only unsure thing about enabling parallel image pulling is if disk will be overloaded.

[ruiwen-zhao]

If GKE will use it， the feedback would be very valuable.

To me， the only unsure thing about enabling parallel image pulling is if disk will be overloaded.

Yep, we (GKE) have done some experiment with different parallelism and observed disk throttling.

One more thing, I don't plan to enable parallel pull by default as part of beta promotion. The beta promotion of this KEP is mostly to tell people this feature is more stable now.

[ruiwen-zhao]

/milestone v1.28

[k8s-ci-robot]

@ruiwen-zhao: You must be a member of the kubernetes/milestone-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Milestone Maintainers Team and have them propose you as an additional delegate for this responsibility.

In response to [this](https://github.com/kubernetes/enhancements/issues/3673#issuecomment-1572536847): >/milestone v1.28 Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[SergeyKanzhelev]

it is already in the milestone

[wojtek-t]

it is already in the milestone

@SergeyKanzhelev It's not in the tracking board https://github.com/orgs/kubernetes/projects/140/views/2 And i'ts because it's marked as Alpha at this point, and is not lead-opted-in for the release. Setting milestone is not enough - you need to lead-opt-in it.

[SergeyKanzhelev]

/label lead-opt-in

[k8s-ci-robot]

@SergeyKanzhelev: The label(s) /label lead-opt-in cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, lead-opted-in, tracked/no, tracked/out-of-tree, tracked/yes. Is this label configured under labels -> additional_labels or labels -> restricted_labels in plugin.yaml?

In response to [this](https://github.com/kubernetes/enhancements/issues/3673#issuecomment-1573972074): >/label lead-opt-in Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[SergeyKanzhelev]

/label lead-opted-in

[SergeyKanzhelev]

/stage beta

[ruheenaansari34]

Hello @ruiwen-zhao 👋, 1.28 Enhancements team here.

Just checking in as we approach enhancements freeze on 1:00 UTC on Friday 16th June 2023.

This enhancement is targeting stage beta for 1.28 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [x] KEP readme using the latest template has been merged into the k/enhancements repo.
• [x] KEP status is marked as implementable for latest-milestone: 1.28
• [ ] KEP readme has a updated detailed test plan section filled out
• [x] KEP readme has up-to-date graduation criteria
• [x] KEP has a production readiness review that has been completed and merged into k/enhancements.

For this KEP, we would need to take care of:

• Updating the Test plan section, especially accepting the acknowledgment specified here

The status of this enhancement is marked as at risk. Please keep the issue description up-to-date with the appropriate stages as well. Thank you!

[ruiwen-zhao]

Hi @ruheenaansari34 , i have sent https://github.com/kubernetes/enhancements/pull/4086 to accept the acknowledgement.

[Atharva-Shinde]

Hey @ruiwen-zhao

With all the KEP requirements in place and merged into k/enhancements, this enhancement is all good for the upcoming enhancements freeze. 🚀 The status of this enhancement is marked as tracked. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[AdminTurnedDevOps]

Hey @ruiwen-zhao

1.28 Docs Shadow here.

Does this enhancement work planned for 1.28 require any new docs or modification to existing docs?

If so, please follows the steps here to open a PR against dev-1.28 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday 20th July 2023.

Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release.

Thank you!

[Rishit-dagli]

Hey @ruiwen-zhao , could you please create a docs PR even if it is a draft PR with no content yet against dev-1.28 branch in the k/website repo. The deadline to create this draft PR is Thursday 20th July 2023.

[ruheenaansari34]

Hey again @ruiwen-zhao 👋 Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

I don't see any code (k/k) update PR(s) in the issue description so if there are any k/k related PR(s) that we should be tracking for this KEP please link them in the issue description above.

As always, we are here to help if any questions come up. Thanks!

[SergeyKanzhelev]

• no update needed as we do not have a feature gates: https://github.com/kubernetes/kubernetes/pull/115220
• beta criteria requires adding the new e2e test. @ruiwen-zhao is out of office. I cannot find the test. @pacoxu have you seen or been working on tests for this?
• docs https://github.com/kubernetes/website/pull/42061 (in case we will have e2e to promote it to beta

[pacoxu]

Just checking in as we approach Code freeze at 01:00 UTC Friday, 19th July 2023 .

@ruheenaansari34 July 19th UTC is Wednesday or Tuesday for some guys. 🤣

• beta criteria requires adding the new e2e test. @ruiwen-zhao is out of office. I cannot find the test. @pacoxu have you seen or been working on tests for this?

@SergeyKanzhelev Let me check. For new e2e test, we may still have a week more for test freeze.

[Atharva-Shinde]

Hello @ruiwen-zhao @SergeyKanzhelev 👋, 1.28 Enhancements Lead here.

With https://github.com/kubernetes/kubernetes/pull/115220 merged as per the issue description, this enhancement is now tracked for v1.28 Code Freeze!

A followup question: Will there be test PRs for this KEP?

Thanks.

[npolshakova]

/remove-label lead-opted-in

[pacoxu]

The website was already reverted for https://github.com/kubernetes/website/pull/42061.

https://github.com/kubernetes/website/blob/main/content/en/docs/concepts/containers/images.md?plain=1#L187

[npolshakova]

Hi @pacoxu, Can you get the lead-opted-in label added to this if it's targeting v1.29? Thanks!

[pacoxu]

Hi @pacoxu, Can you get the lead-opted-in label added to this if it's targeting v1.29? Thanks!

@npolshakova we will have a KEP planning sig meeting today for the sig node.

@SergeyKanzhelev Will lead-opted-in label be added in the SIG meeting?

[SergeyKanzhelev]

/label lead-opted-in

[salehsedghpour]

Hello @ruiwen-zhao and @pacoxu 👋, Enhancements team here.

Just checking in as we approach enhancements freeze on Friday, 6th October 2023.

This enhancement is targeting for stage stable for 1.29 (correct me, if otherwise)

Here's where this enhancement currently stands:

• [ ] KEP readme using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implemented for latest-milestone: 1.29.
• [ ] KEP readme has up-to-date graduation criteria
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements. (For more information on the PRR process, check here).

For this KEP, we would just need to add/update the following:

• KEP readme using the latest template has been merged into the k/enhancements repo
• The kep.yaml file based on the recent changes
• Update the issue description.

The status of this enhancement is marked as at risk for enhancement freeze. Please keep the issue description up-to-date with appropriate stages as well. Thank you!

[pacoxu]

This enhancement is targeting for stage stable for 1.29 (correct me, if otherwise)

@salehsedghpour https://github.com/kubernetes/enhancements/pull/4173 is opened. We are targeting for stage beta. For v1.28 release cycle, the kep and website PR are merged in exception without adding some new e2e test.

• In v1.29, we need to re-promote it to beta with new e2e test. (There is no feature gate here)

[salehsedghpour]

@pacoxu, many thanks for the clarification. The only thing that is missing is the README file. Recently some questions added in the production readiness review questionnaire such as this. I would appreciate it if you can make sure that the readme file is updated.

[pacoxu]

@pacoxu, many thanks for the clarification. The only thing that is missing is the README file. Recently some questions added in the production readiness review questionnaire such as this. I would appreciate it if you can make sure that the readme file is updated.

@salehsedghpour If you mean the change from https://github.com/kubernetes/enhancements/pull/4137, this will not be effected as this is a node-only feature. It will not be related to API server or version skew policy. I will update the KEP later.

[salehsedghpour]

Hi @ruiwen-zhao and @pacoxu , checking in once more as we approach the 1.29 enhancement freeze deadline on 01:00 UTC, Friday, 6th October, 2023. The status of this enhancement is marked as at risk. It looks like https://github.com/kubernetes/enhancements/pull/4173 addressed most of the requirements.

@pacoxu, the main change is the following question that is added in the recent README template: Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

Let me know if I missed anything. Thanks!

[ruiwen-zhao]

Hi @ruiwen-zhao and @pacoxu , checking in once more as we approach the 1.29 enhancement freeze deadline on 01:00 UTC, Friday, 6th October, 2023. The status of this enhancement is marked as at risk. It looks like #4173 addressed most of the requirements.

@pacoxu, the main change is the following question that is added in the recent README template: Can enabling / using this feature result in resource exhaustion of some node resources (PIDs, sockets, inodes, etc.)?

Let me know if I missed anything. Thanks!

thanks @salehsedghpour ! I created https://github.com/kubernetes/enhancements/pull/4274 to add that question to the KEP.

[salehsedghpour]

@ruiwen-zhao, thanks, with that being merged this enhancement can be marked as tracked. I'll keep eye on the #4274 so that I can update the status.

[salehsedghpour]

As #4274 is merged now, the status of this enhancement is now marked as tracked for enhancement freeze.

[drewhagen]

Hello @pacoxu @ruiwen-zhao :wave:, v1.29 Docs Shadow here. Does this enhancement work planned for v1.29 require any new docs or modification to existing docs? If so, please follows the steps here to open a PR against dev-1.29 branch in the k/website repo. This PR can be just a placeholder at this time and must be created before Thursday, 19 October 2023. Also, take a look at Documenting for a release to get yourself familiarize with the docs requirement for the release. Thank you!

[pacoxu]

No doc update this time: https://github.com/kubernetes/website/pull/42061.

• we should probably revert the pr for v1.28 as the e2e test has not been added yet.

I plan to work on this in the following weeks to add some tests.