search

kubernetes / kops

Kubernetes Operations (kOps) - Production Grade k8s Installation, Upgrades and Management
https://kops.sigs.k8s.io/
Apache License 2.0
15.66k stars 4.61k forks source link

Have periodic E2E jobs use the same branch for kops and kubetest2-kops #16084

Open rifelpet opened 8 months ago

rifelpet commented 8 months ago

/kind feature

1. Describe IN DETAIL the feature/behavior/change you would like to see.

Currently all E2E jobs use the HEAD of master for building kubetest2-kops, regardless of the kops branch being tested.

This was convenient during initial kubetest2-kops development, to make sweeping changes that impact all jobs.

The downside of this is when kubetest2-kops changes require backwards-incompatible changes to kops and its release process. We need to cherrypick changes to kops and/or kubetest2-kops to ensure all tested versions remain in sync.

For example, this job is failing because something changed with the kops staging bucket location in master, but it hasn't been cherrypicked back to all tested kops versions.

Additionally, newer kubetest2 versions only support testing k8s versions newer than kops' current grid.

For example, this job uses kops 1.27 to test k8s 1.24 and is currently failing because k8s 1.24's e2e suite uses ginkgo v1, but the master branch of kubetest2-kops depends on a version of kubetest2 that requires ginkgo v2

 Copying gs://kubernetes-release/release/v1.24.17/kubernetes-test-linux-amd64.tar.gz...
/ [0 files][    0.0 B/114.2 MiB]                                                
-
- [1 files][114.2 MiB/114.2 MiB]                                                
\
Operation completed over 1 objects/114.2 MiB.                                    
Copying gs://kubernetes-release/release/v1.24.17/bin/linux/amd64/kubectl...
/ [0 files][    0.0 B/ 44.5 MiB]                                                
/ [1 files][ 44.5 MiB/ 44.5 MiB]                                                
Operation completed over 1 objects/44.5 MiB.                                     
F1106 02:16:51.700377   13960 tester.go:505] failed to run ginkgo tester: unsupported ginkgo version: 1

If we instead used the kops 1.27 branch for kubetest2-kops, we could keep it on an older kubetest2 dependency version that still supports ginkgo v1.

2. Feel free to provide a design supporting your feature request.

I think we could achieve this with a couple steps:

  1. Cherrypick almost all kubetest2-kops commits on master back to all supported kops release branches. This may get messy with the kubetest2 dependency. Other go dependency changes shouldn't be a problem because of our isolated test/e2e go module.

  2. Update our build_jobs.py template to checkout the same kops branch for building and running kubetest2-kops as the jobs' kops version. This may also require updating the kubekins-e2e image used in the job to use the associated k8s release branch.

rifelpet commented 7 months ago

We will do this going forward, starting with the release-1.29 kops branch. This should only require changes to the build_jobs.py script, no kubetest2-kops cherry-picking for the initial switch.

This means once release-1.29 is created, future kubetest2-kops changes will need to be cherry-picked back to release-1.29

k8s-triage-robot commented 4 months ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

rifelpet commented 4 months ago

/remove-lifecycle stale

k8s-triage-robot commented 1 month ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

rifelpet commented 1 month ago

/remove-lifecycle stale