kubetest2 CI migration

[amwat]

Enhancement Description

• One-line enhancement description (can be used as a release note): Migration of presubmit-blocking and release-blocking end-to-end CI jobs to kubetest2

• Kubernetes Enhancement Proposal: https://git.k8s.io/enhancements/keps/sig-testing/2464-kubetest2-ci-migration

• Discussion Link:

• Primary contact (assignee): @amwat

• Responsible SIGs: SIG-Testing, SIG-Release

• Enhancement target (which target equals to which milestone):

  • Alpha release target (x.y): 1.21
  • Beta release target (x.y): 1.23
  • Stable release target (x.y): 1.24

• [x] Alpha

  • [x] KEP (k/enhancements) update PR(s): https://github.com/kubernetes/enhancements/pull/2465, https://github.com/kubernetes/enhancements/pull/2479
  • [x] Code (k/k) update PR(s): N/A
  • [x] Docs (k/website) update PR(s): N/A

• [ ] Beta

  • [x] KEP (k/enhancements) update PR(s): https://github.com/kubernetes/enhancements/pull/2871
  • [x] Code (k/k) update PR(s): https://github.com/kubernetes/kubernetes/pull/106013 , https://github.com/kubernetes/kubernetes/pull/105637
  • [x] Docs (k/website) update(s): N/A

Please keep this description up to date. This will help the Enhancement Team to track the evolution of the enhancement efficiently.

This will be in: https://github.com/kubernetes/enhancements/tree/master/keps/sig-testing/2464-kubetest2-ci-migration

/assign /sig testing /sig release

[spiffxp]

/milestone v1.21

[spiffxp]

/priority important-soon

[BenTheElder]

This will be merged implementable momentarily #2479

[spiffxp]

I would bucket this KEP under process when https://github.com/kubernetes/enhancements/issues/2311 is completed

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

[fejta-bot]

Enhancement issues opened in kubernetes/enhancements should never be marked as frozen. Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

/remove-lifecycle frozen

[BenTheElder]

Enhancement Owners can ensure that enhancements stay fresh by consistently updating their states across release cycles.

Release cycles are now 4 months long. I don't believe this is accurate / the most reasonable anymore. fejta-bot stales an issue after 90 days (rotten + closed after 30 more), but release cycles are now roughly 1/3 a year ...

[spiffxp]

I'm adding a tracked/no label to this as a best guess but I'm open to suggestion from @kubernetes/enhancements or @kubernetes/release-team

It's the sort of thing that either lands prior to Test Freeze, or it doesn't, since this determines what is being used for CI signal

[spiffxp]

/milestone v1.23

[spiffxp]

/stage beta https://github.com/kubernetes/enhancements/pull/2871 proposes moving this to beta in v1.23

[Priyankasaggu11929]

Hi @amwat! 1.23 Enhancements team here. Just checking in as we approach enhancements freeze on Thursday 09/09. Here's where this enhancement currently stands:

• [ ] KEP file using the latest template has been merged into the k/enhancements repo.
• [ ] KEP status is marked as implementable
• [ ] KEP has a test plan section filled out.
• [X] KEP has up to date graduation criteria.
• [ ] KEP has a production readiness review that has been completed and merged into k/enhancements.

Looks like for this one, we would just need the following:

• update the PR https://github.com/kubernetes/enhancements/pull/2871 to fill the "Test Plan" section in the KEP README.md file
• and have the PR merged, to check the PRR & the status checkbox.

Thanks!

[Priyankasaggu11929]

Thanks so much for the changes, @amwat. With the KEP PR merged, this enhancement is ready for 1.23 enhancements freeze. :)

[spiffxp]

/kind kep

[spiffxp]

Checklist for beta:

• [ ] a guide for migration from kubetest to kubetest2 exists
• [ ] gce jobs have moved to kubetest2 (release-blocking, merge-blocking)
  • [ ] ci-kubernetes-e2e-gce-device-plugin-gpu
  • [ ] ci-kubernetes-e2e-gce-master-new-gci-kubectl-skew
  • [ ] ci-kubernetes-e2e-gci-gce
  • [ ] ci-kubernetes-e2e-gci-gce-alpha-features
  • [ ] ci-kubernetes-e2e-gci-gce-ingress
  • [ ] ci-kubernetes-e2e-gci-gce-reboot
  • [ ] ci-kubernetes-e2e-ubuntu-gce-containerd
  • [ ] pull-kubernetes-e2e-gce-network-proxy-http-connect
  • [ ] pull-kubernetes-e2e-gce-ubuntu-containerd
• [ ] release-blocking/merge-blocking node jobs have corresponding shadows on k/k master
  • [ ] ci-kubernetes-node-kubelet
  • [ ] pull-kubernetes-node-e2e: https://github.com/kubernetes-sigs/kubetest2/issues/165
  • [ ] pull-kubernetes-node-e2e-containerd
• [ ] release-blocking/merge-blocking scale jobs have corresponding shadows on k/k master
  • [ ] ci-kubernetes-e2e-gci-gce-scalability
  • [ ] pull-kubernetes-e2e-gce-100-performance

[Priyankasaggu11929]

Hello @amwat 👋

Checking in once more as we approach 1.23 code freeze at 6:00 pm PST on Tuesday, November 16.

Please ensure the following items are completed:

• All PRs to the Kubernetes repo that are related to your enhancement are linked in the above issue description (for tracking purposes).
• All PRs are fully merged by the code freeze deadline.

As always, we are here to help should questions come up.

Thank you so much! 🙂

[amwat]

Thanks @Priyankasaggu11929 , updated the PR description with latest info.

[Priyankasaggu11929]

Thank you so much for the update @amwat. 🙂

[spiffxp]

@amwat did we complete any of the checklist items to actually graduate to beta? I saw a lot of shadow presubmits added this cycle but I'm less clear whether there are any more blocking jobs using kubetest2 than last release

[amwat]

Yeah, all of those are are towards the 3rd beta requirement of having shadow node jobs.

We have an overall community guide for migration at: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-testing/e2e-tests-kubetest2.md#testing-against-an-existing-cluster I think we can keep adding FAQs for migrating specific types of jobs to this doc, but that probably also extends beyond the limitations of this KEP.

But we have not yet made pull-kubernetes-e2e-gce-kubetest2 blocking (thanks for the reminder!)

Note sure if we want to update the target accordingly then.

[Priyankasaggu11929]

Hello @amwat, could you help providing a status summary on the checklist, mentioned in one of the above comments here. Want to confirm if all fo the items are checked or if not, what all are still in progress..

Thank you so much! 🙂

[SergeyKanzhelev]

@amwat what is the status of this? Is there plan to fully transition sig node tests to kubetest2 in 1.24?

[BenTheElder]

@SergeyKanzhelev amwat moved on before your comment and doesn't work in Kubernetes anymore. I don't know who can take up the remainder yet.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[Namanl2001]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[cpanato]

/remove-lifecycle stale

[cpanato]

i will take a look on this, maybe just need some quick recap :) cc @BenTheElder

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[cpanato]

@BenTheElder gently ping here :)

[BenTheElder]

@cpanato I don't have a lot to add unfortunately, the last I saw we were blocked on getting SIG Node input to migrate node_e2e support, and the large lift to port existing job configs otherwise, in theory the tooling works for kube-up/kOps e2e tests today and it is used for kOps and some kube-up.sh/GCE jobs.

[BenTheElder]

I stepped down from kubetest/kubetest2, it became one too many projects to maintain with the high focus on infra costs. I'm no longer an approver in kubetest2.

cc @aojea @pohly @cjwagner @alvaroaleman (the rest of our SIG TLs)

[aojea]

If nobody is going to work on this we should close the issue,no?

is not really a feature, it seems to track only the effort to do it

[BenTheElder]

@aojea https://github.com/kubernetes/enhancements/issues/2464#issuecomment-1235407789 ?

otherwise yes

[vaibhav2107]

/remove-lifecycle stale

[cpanato]

agree @aojea

[upodroid]

/assign

Can we track this for 1.29?

Core Goals:

• Run the node e2e tests using kubetest2 and replace the old kubetest1 tests that are part of master-blocking/informing
• ~Run the GCE Kubernetes e2e tests against amd64 on kubetest2 and replace the old kubetest1 tests that are part of master-blocking/informing~
• Roll out the EC2 node tests using kubetest2

~Stretch Goals:~

• ~arm64 cluster e2e tests on GCE. Initial work https://github.com/kubernetes/kubernetes/pull/120144~

@dims

[dims]

@upodroid sounds good to me :)