setup release 1.9 branch

BenTheElder commented 6 years ago

The branch was cut tonight, I'm working on getting all the right configs updated and figured it would be useful to collect all the PRs involved against this issue for next time.

/area jobs /area testgrid

BenTheElder commented 6 years ago

Derailed for the moment by looking into https://github.com/kubernetes/kubernetes/issues/55917, I suspect that one will not have a great simple fix, it looks like multiple PRs changing the versioning logic went in recently.

enisoc commented 6 years ago

ref https://github.com/kubernetes/sig-release/issues/30

BenTheElder commented 6 years ago

Moving the checklist here and breaking up the PR, we decided to do it piece-meal and put in the build job(s) first:

To setup release-1.9:

Testgrid:

[x] create sig-release-1.9-all dashboard
[x] create sig-release-1.9-blocking dashboard
[x] [WIP] add entries for all release-1.9 jobs to the above dashboards
[x] add 1.9 jobs to any other appropriate dashboards
[x] [WIP] sig dashboards for sig relatedrelease-1.9 jobs
[x] move appropriate jobs into the blocking dashboard (~ match release-1.8)

Jenkins:

[x] add ci-kubernetes-build-1.9, bump stableN for build jobs
[x] add kubernetes-test-go-release-1.9
[x] add kubernetes-verify-release-1.9
[x] update kubernetes-kubemark-5-gce-last-release trigger job to ci-kubernetes-build-1.9
[ ] take a second pass to check for any remaining jobs

Prow:

[x] ensure presubmits will run against release-1.9 branch the same as existing master
[x] add 1.9 versions of bazel ci/periodic jobs
[x] add 1.9 versions of kubeadm ci/periodic jobs
[x] add 1.9 versions of device-plugin / gpu jobs
[ ] add 1.9 versions of federation jobs (punting to sig-multicluster, see comments below)
[x] setup 1.9 versions kubectl-skew jobs

Other TODO:

[x] generate jobs where possible (@krzyzacy) #5573
[ ] [WIP] move jobs to Prow where possible, xref #3778
- Need to add more resources to prow-builds, also https://github.com/kubernetes/test-infra/pull/5561
- 5585

/area jobs

BenTheElder commented 6 years ago

@shashidharatd @marun what's the deal with kubernetes-federation-build-1.8 etc.?

BenTheElder commented 6 years ago

Complete list of merged PRs involved (will keep up to date):

Add release 1.9 build (what it says) https://github.com/kubernetes/test-infra/pull/5567
add relase-1.9 verify and test-go (also fix dashboard grouping, but apparently not PR title spelling) https://github.com/kubernetes/test-infra/pull/5569
Branch bikeshed (generate jobs) https://github.com/kubernetes/test-infra/pull/5573 by @krzyzacy
update kubernetes-kubemark-5-gce-last-release trigger to 1.9 instead … (update kubemark ci job trigger and kubernetes version) https://github.com/kubernetes/test-infra/pull/5574
add missing config.json entries for test-go and verify jobs for relea… https://github.com/kubernetes/test-infra/pull/5580
add release 1-9 ci/periodic bazel jobs (also kubeadm) https://github.com/kubernetes/test-infra/pull/5576
Add stable1-beta (currently 1.8-1.9) upgrade/skew jobs https://github.com/kubernetes/test-infra/pull/5609 by @krzyzacy
add gpu jobs for 1.9 https://github.com/kubernetes/test-infra/pull/5616
Add kops-beta job https://github.com/kubernetes/test-infra/pull/5635 by @krzyzacy
Add kubelet and scalability suite for beta, and add to 1.9 dashboard https://github.com/kubernetes/test-infra/pull/5649 by @krzyzacy
add gpu-p100 job to release 1.9 https://github.com/kubernetes/test-infra/pull/5661
disable integration tests ci-kubernetes-bazel-test to match presubmits https://github.com/kubernetes/test-infra/pull/5664
Release 1.9 bazel patchup https://github.com/kubernetes/test-infra/pull/5680
fix incorrect flag in ci-kubernetes-test-go-1.9 https://github.com/kubernetes/test-infra/pull/5684
kick kubeadm upgrade job out of release-1.9-blocking https://github.com/kubernetes/test-infra/pull/5724
fix release 1.9 bazel build job branch https://github.com/kubernetes/test-infra/pull/5728

Semi related PRs:

migrate pull-kubernetes-cross to prow https://github.com/kubernetes/test-infra/pull/5585
Change dashboard names to use 1.y instead of 1-y https://github.com/kubernetes/test-infra/pull/5648 by @spiffxp
add bazel jobs to release-master-blocking https://github.com/kubernetes/test-infra/pull/5654
disable integration tests ci-kubernetes-bazel-test to match presubmits https://github.com/kubernetes/test-infra/pull/5664

shashidharatd commented 6 years ago

@shashidharatd @marun what's the deal with kubernetes-federation-build-1.8 etc.?

now federation is moved to k/f repo and the ci build for head is here https://k8s-testgrid.appspot.com/sig-multicluster-federation-gce#build

To create a federation-build release against 1.9 we need to use the jobs which are running against the master branch. Also there are some specific requirements(like increased quota) for federation e2e jobs.

Definetely the federation build is a non-blocker for k8s 1.9 release. We still need to conclude the federation release timelines, whether it should be inline with k8s releases or should have independent release which works for the corresponding k8s release. We will discuss this in our SIG meeting on tuesday.

/cc @kubernetes/sig-multicluster-pr-reviews

BenTheElder commented 6 years ago

I think this is done minus the federation jobs. Edit and kops :^) (Fixed)

spiffxp commented 6 years ago

A few nits while attempting to reconcile with diff. In an ideal world, release-1.y-blocking would just be a copy of release-master-blocking with jobs pointed at different branches. Trying to understand why the differences exist. I'm hoping there are some common explanations which we could add in comment form someplace, eg:

was added as a release-blocker during v1.8
can only be run against release-1.y branches
etc.

@kubernetes/sig-gcp-test-failures The gci-{gce,gke} names in sig-release-{master,1.8} are now just {gce,gke} in sig-release-1.9-blocking, can we strip the gci- prefix from the other names? There are both gce and gci-gce jobs in sig-release-1.7-blocking

~@kubernetes/sig-testing-test-failures These jobs in sig-release-1.9-blocking have no equivalent in sig-release-{master,1.8,1.7}-blocking. Should they be removed from release-1.9 or added to any of the others?~ these were added during the v1.9 dev cycle, #5654 ensure they're in master, and we're not backporting

~bazel-build-1.9~
~bazel-test-1.9~

@kubernetes/sig-cluster-lifecycle-test-failures These jobs in sig-release-1.9-blocking have no equivalent in sig-release-{master,1.8}-blocking. Should they be removed from release-1.9 or added to any of the others?

gce-kubeadm-upgrade-1.8-1.9

@kubernetes/sig-multicluster-test-failures These jobs in sig-release-1.8-blocking have no equivalent in sig-release-{master,1.9,1.7}-blocking. Should they be removed from release-1.8 or added to any of the others?

federation-build-1.8
federation-test-1.8

~@kubernetes/sig-network-test-failures These jobs in sig-release-{master,1.9}-blocking have no equivalent in sig-release-{1.8,1.7}-blocking. Should they be added to any of the others?~ disregard, @spiffxp was blind

~gce-ingress-1.9~
~gke-ingress-1.9~

@kubernetes/sig-scalability-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-{1.9,1.8,1.7}-blocking. Should they be added to any of the others?

gce-scale-correctness
gce-scale-performance

~@kubernetes/sig-auth-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-{1.9,1.8,1.7}-blocking. Should they be added to any of the others?~ The goal is to fold these tests into the standard suite, and remove the gci-gce-audit job (ref: kubernetes/kubernetes#55953)... and the job is removed via #5690

~gci-gce-audit~

~@kubernetes/sig-scheduling-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-{1.9,1.8,1.7}-blocking. Should they be added to any of the others?~ I'm going to claim this was added during the v1.9 dev cycle, and we've added it to the 1.9 board via #5661

~gke-device-plugin-gpu-p100~

~@kubernetes/sig-node-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-1.9-blocking. Should they be added to any of the others?~ #5649 added kubelet-1.9

~kubelet~

misc:

~gci-gce-100 (test_group_name is ci-kubernetes-e2e-gci-gce-scalability, there is a gce-scalability job in 1.8 and 1.7)~ this has been added
~soak-gce-gci~ this was dropped from master via #5529

krzyzacy commented 6 years ago

For the ones I know:

I have blown away all the 1.7-gke-'cvm' jobs
@BenTheElder has added the bazel jobs, we'll need to add that to 1.9-blocking dash
we don't create separate gce-scale jobs for release (they are expensive!) but we probably should have a gci-gce-100 for 1.9, I'll create it.
I'll add a kubelet job
Do we still want a soak job? I'll create one for sure.

BenTheElder commented 6 years ago

+1 to the comments on why something exists, I've at least tried to make sure all the changes we make for 1.9 show up https://github.com/kubernetes/test-infra/issues/5558#issuecomment-345392515

yujuhong commented 6 years ago

@kubernetes/sig-node-test-failures These jobs in sig-release-master-blocking have no equivalent in sig-release-1.9-blocking. Should they be added to any of the others?

kubelet

Yes, we have kubelet-1.8, and kubelet 1.7 jobs.

krzyzacy commented 6 years ago

added kubelet ones from https://github.com/kubernetes/test-infra/pull/5649

BenTheElder commented 6 years ago

@BenTheElder has added the bazel jobs, we'll need to add that to 1.9-blocking dash

These jobs block on presubmit, and are no more flaky than unit / non-bazel build. They're in the 1.9 blocking dash but we should have them in master blocking too.

spiffxp commented 6 years ago

has added the bazel jobs, we'll need to add that to 1.9-blocking dash

Yeah, they're there, I guess my question to @BenTheElder is, is there any reason they're blocking 1.9 but not blocking master?

lol ok question answered above, as a member of the release team I'll lgtm a pr that adds them based on history of passing against PR's

BenTheElder commented 6 years ago

I'm adding them to master. We have no reason to not block on these.

BenTheElder commented 6 years ago

Federation jobs for 1.9 have been punted to federation to sort out how they want them but "the federation build is a non-blocker for k8s 1.9 release" (since they moved out of k/k?) https://github.com/kubernetes/test-infra/issues/5558#issuecomment-345624723

BenTheElder commented 6 years ago

For sig questions about blocking tests in comment above (https://github.com/kubernetes/test-infra/issues/5558#issuecomment-346385580) I am asking the following sig members to respond with how they would like these issues handled :-)

cc @enisoc @abgworrall for possible GCP job name bikeshed
cc @luxas for kubeadm / sig-cluster-lifecycle jobs potentially missing from master and previous releases
@shashidharatd discussed federation / sig-multicluster above though this isn't solved yet it shouldn't be a problem for now
cc @MrHohn for sig-network jobs missing from previous releases
not sure who to flag down for sig-auth, but the sigs have all been pinged at their test failures handle anyhow..
cc @mindprince for sig-scheduling / GPU jobs only present in master blocking but not any of the release branches (including 1.9)

Please let us know what makes sense for these questions and I will make the necessary changes.

rohitagarwal003 commented 6 years ago

These jobs in sig-release-master-blocking have no equivalent in sig-release-{1.9,1.8,1.7}-blocking. Should they be added to any of the others? gke-device-plugin-gpu-p100

Yes. Let's add them to release-1.9-blocking. When I added that job, release-1.9-blocking didn't exist, so I only added it to master-blocking.

BenTheElder commented 6 years ago

Yes. Let's add them to release-1.9-blocking. When I added that job, release-1.9-blocking didn't exist, so I only added it to master-blocking.

Thanks, we'll do that :+1:

MrHohn commented 6 years ago

kubernetes/sig-network-test-failures These jobs in sig-release-{master,1.9}-blocking have no equivalent in sig-release-{1.8,1.7}-blocking. Should they be added to any of the others?

gce-ingress-1.9

gke-ingress-1.9

@BenTheElder I think we do have equivalent jobs, just that the naming format changed?

spiffxp commented 6 years ago

@MrHohn you're right, that was my bad, not sure how I missed those

BenTheElder commented 6 years ago

Thanks @MrHohn, with all of the jobs I don't have full track of every job 🙃

BenTheElder commented 6 years ago

This https://github.com/kubernetes/test-infra/issues/5663 / the associated PR https://github.com/kubernetes/test-infra/pull/5664 will be relevant now that https://github.com/kubernetes/kubernetes/pull/55279 is in.

crassirostris commented 6 years ago

@spiffxp Audit tests will be added to the default suite in gce/gke env by https://github.com/kubernetes/kubernetes/pull/55953

spiffxp commented 6 years ago

@crassirostris does this mean we should remove the audit-specific job once that PR merges?

crassirostris commented 6 years ago

@spiffxp Yes

BenTheElder commented 6 years ago

There are a few unanswered questions above, but the branch has been set up and running for some time (with a few fixes since then), I think we should close this.

kubernetes / test-infra

setup release 1.9 branch #5558

5585