search

kubernetes / sig-release

Repo for SIG release
Apache License 2.0
543 stars 392 forks source link

Assess version markers used in release-branch job configs #850

Open justaugustus opened 5 years ago

justaugustus commented 5 years ago

Version markers are text files stored in the root of various GCS buckets:

They represent the results of different types of Kubernetes build jobs and act as sort of a public API for accessing builds. One can see them leveraged in extraction strategies for e2e tests, release engineering tooling, and user-created scripts.

Unfortunately, the way certain version markers are generated and utilized can at best be confusing, and at worst, disruptive.

There are a variety of problems, some of which are symptoms of the other ones...

Generic version markers are not explicit

We publish a set of additional generic version markers:

Depending on the point in the release cycle, the meaning of these markers can change.

Knowing what these markers mean at any one time presumes knowledge of the build/release process or a correct interpretation of the Kubernetes versions doc, which has frequently been out of date and lives in a low-visibility location.

Manually created jobs using generic version markers can be inaccurate

Non-generated jobs using generic version markers do not get the same level of scrutiny as ones that are generated via releng/test_config.yaml.

This leads to inaccuracies between the versions presumed to be used in test and the versions that may be displayed in testgrid.

ci-kubernetes-e2e-gce-beta-stable1-gci-kubectl-skew is a great example:

https://github.com/kubernetes/test-infra/blob/96e08f4be2a86189f59c72055785f817ac346d30/config/jobs/kubernetes/sig-cli/sig-cli-config.yaml#L85-L112

All variants of that prowjob have landed on the sig-release-job-config-errors dashboard for various misconfiguration issues that are the result of generic version markers.

I'd like to establish a rough plan of record to continue iteratively fixing some of these issues.

Plan of record

Previous Issues

linux/amd64 version markers are colliding with cross builds

(Fixed in https://github.com/kubernetes/test-infra/pull/18290.)

"Fast" (linux/amd64-only) builds run every 5 minutes, while cross builds run every hour. They also write to the same version markers (latest.txt, latest-<major>.txt, latest-<major>.<minor>.txt).

The Kubernetes build jobs have a mechanism for checking if a build already exists and will exit early to save on test cycles.

What this means is if a "fast" build has already happened for a commit, then the corresponding cross build will exit without building.

This has been happening pretty consistently lately, so cross build consumers are using much older versions of Kubernetes than intended.

(Note that this condition only happens on master.)

Cross builds are stored in a separate GCS bucket

(Fixed in https://github.com/kubernetes/test-infra/pull/14030.)

This makes long-term usage of cross builds a little more difficult, since scripts utilizing version markers tend to consider only the version marker filename, while the GCS bucket name remains unparameterized.

Generated jobs may not represent intention

(Fixed in https://github.com/kubernetes/test-infra/pull/15564.)

As the generic version markers can shift throughout the release cycle, every time we regenerate jobs, they may not represent what we intend to test.

The best examples of this are pretty much every job using the k8s-beta version marker, and more specifically, skew and upgrade jobs.

bazel version markers appear to be unused

(Fixed in https://github.com/kubernetes/test-infra/pull/15612.)

ref: https://github.com/kubernetes/test-infra/pull/15106

/assign /area release-eng /priority important-longterm /milestone v1.17

tpepper commented 5 years ago

/cc

justaugustus commented 4 years ago

To respond to @spiffxp's comment on the Branch Management issue:

I would suggest that branch management is the role that should handle "what 'channel' (beta/stable1/stable2/stable3) corresponds to which version?"

Agreed. Now codified in the Branch Management handbook.

  • the beta channel should only exist after the release-1.y branch is cut, and be unused after the v1.y.0 release is cut (aka during the period that builds being cut have the word beta in them, and the branch manager is running branchff and handling cherry-picks prior to the .0 release)
  • the stableN versions are moved forward after the v1.y.0 release is cut, so that stable1 refers to v1.y.0, the most recent stable release, stable2 refers to v1.y-1.0, the previous stable release, etc.
  • release teams have forgotten to do this last part since 1.11 (ref: ref: kubernetes/test-infra#13577 (comment)), so we're in a state where the channels don't mean what they should

other ideas include:

I liked this idea and actually mentioned this to @tpepper after bumping into https://github.com/kubernetes/test-infra/issues/15514.

My thought here is that creating the new release branch jobs immediately after the final patch release would result in turning down CI on the last-supported branch way sooner and giving time to watch the new release branch jobs stability.

This comes at the cost of more branch fast-forwards and more cherry picks.

Do we think that's worth it?

I'm not a fan of this nomenclature, especially because it has consistently caused confusion and inconsistency around what's under test at one period in the release cycle.

Are there any glaring things that we'd need to look out for going down this route?

spiffxp commented 4 years ago

This comes at the cost of more branch fast-forwards and more cherry picks.

I don't think it would cause more cherry-picks? Those don't start happening until after code freeze.

I was envisioning that alpha's would get cut off of the release-1.y branch with this approach, and that master's version wouldn't bump until after code freeze. This is different than today, where master's version bumps as soon as the release branch is cut.

Are there any glaring things that we'd need to look out for going down this route?

We'll "lose" historical data for jobs on our dashboards (testgrid, triage, velodrome, etc), since none of them comprehend job renames or moves. Early in the release cycle is probably the best time to induce such a gap.

Outside of that I suspect it's not glaring things, just lots of tiny renames. @Katharine might be able to better explain what prevented us from moving ahead with the rename in https://github.com/kubernetes/test-infra/pull/12516.

justaugustus commented 4 years ago

I don't think it would cause more cherry-picks? Those don't start happening until after code freeze.

@spiffxp -- Good point. This was mushy brain from triaging other stuff.

We'll "lose" historical data for jobs on our dashboards (testgrid, triage, velodrome, etc), since none of them comprehend job renames or moves. Early in the release cycle is probably the best time to induce such a gap.

I think I'm fine with losing some historical data if it leads to ease of management for the team over time.

@kubernetes/release-engineering -- What are your thoughts on this?

justaugustus commented 4 years ago

Some discussion in Slack here: https://kubernetes.slack.com/archives/C09QZ4DQB/p1576104279099300 ...and I'm poking at the version markers and release job generation here: https://github.com/kubernetes/test-infra/pull/15564

justaugustus commented 4 years ago

Here's another instance of wrestling with version markers being a general nightmare: https://github.com/kubernetes/test-infra/pull/15875

That PR should've been at most a few commits.

This cycle I'm going to be looking at renaming the release-branch jobs that reference beta,stable{1,2,3} and removing the generic suffix annotations. That should be an easy-ish way to get started.

From there, we'll need to look at refactoring generate_tests.py and test_config.yaml.

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

fejta-bot commented 4 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle rotten

fejta-bot commented 4 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /close

k8s-ci-robot commented 4 years ago

@fejta-bot: Closing this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/850#issuecomment-642535223): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. > >Send feedback to sig-testing, kubernetes/test-infra and/or [fejta](https://github.com/fejta). >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
justaugustus commented 4 years ago

PR to support uploading multiple additional version markers in push-build.sh (ci-kubernetes-build* jobs): https://github.com/kubernetes/release/pull/1385

justaugustus commented 4 years ago

@BenTheElder @spiffxp -- Updated the issue description with more details and a plan of record.

/kind bug cleanup /milestone v1.19 /remove-priority important-longterm

/priority critical-urgent (as this is impacting users who rely on the cross build markers)

spiffxp commented 4 years ago

I'd like to make sure we unblock the cross build stuff first. A checklist of work is a good start. I still think we're lacking a description of the desired state. My guess is the desired state involves no more "generic" version markers. I'll post more detailed review when I get time later today.

lasomethingsomething commented 4 years ago

Heya, any updates on this since @spiffxp's last comment? Based on chats this week with key Release Team members this remains high-priority. Anything we could delegate here?

neolit123 commented 4 years ago

i think we should have something like:

ci/latest
ci/latest-x
ci/latest-x.yy
release/stable
release/stable-x
release/stable-x.yy

and that's all that we need...not sure why the project would need anything else?

markers with -fast suffix seems fine for k8s maintainers, but for external consumption it does not make much sense. IMO, we should remove all k8s- markers.

skew test jobs for e.g. kubectl could use ci/latest-x.yy explicitly instead of markers that imply N-1, N-2. SIG CLI could upgrade this on each cycle or we could have automation in test-infra to update the jobs.

for the kubeadm jobs today we only use ci/latest* and we update them each cycle.

BenTheElder commented 4 years ago

Most of CI should be consuming quick builds. If I wanted to support CI builds in a tool like kind I'd want the fast builds when possible. The slow builds are slow.

Oppositely, most things don't really need to slow builds. We have little to no CI that actually depend on these.

On Wed, Aug 5, 2020 at 1:16 PM Lubomir I. Ivanov notifications@github.com wrote:

i think we should have something like:

ci/latest ci/latest-x ci/latest-x.yy release/stable release/stable-x release/stable-x.yy

and that's all that we need...not sure why the project would need anything else?

markers with -fast suffix seems fine for k8s maintainers, but for external consumption it does not make much sense. IMO, we should remove all k8s- markers.

skew test jobs for e.g. kubectl could use ci/latest-x.yy explicitly instead of markers that imply N-1, N-2. SIG CLI could upgrade this on each cycle or we could have automation in test-infra to update the jobs.

for the kubeadm jobs today we only use ci/latest* and we update them each cycle.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/sig-release/issues/850#issuecomment-669479097, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHADKYR4Q7WV3CN6RREHBDR7G4ZDANCNFSM4JJGWGLA .

justaugustus commented 4 years ago

Heya, any updates on this since @spiffxp's last comment?

Yep. Chatted w/ Aaron and the description of the issue reflects his request now.

Based on chats this week with key Release Team members this remains high-priority. Anything we could delegate here?

There's still a bit of cleanup to do here on my part around the generic version markers.

i think we should have something like:

ci/latest
ci/latest-x
ci/latest-x.yy
release/stable
release/stable-x
release/stable-x.yy

and that's all that we need...not sure why the project would need anything else?

There are also the release/latest**, which represents pre-release versions. https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md was updated recently to discuss the markers currently in use.

markers with -fast suffix seems fine for k8s maintainers, but for external consumption it does not make much sense.

+100

IMO, we should remove all k8s- markers.

Already part of the plan, described under Generic version markers are not explicit.

skew test jobs for e.g. kubectl could use ci/latest-x.yy explicitly instead of markers that imply N-1, N-2. SIG CLI could upgrade this on each cycle or we could have automation in test-infra to update the jobs.

Agreed. Described in Manually created jobs using generic version markers can be inaccurate and part of Refactor any non-generated jobs using generic version markers.

Most of CI should be consuming quick builds. If I wanted to support CI builds in a tool like kind I'd want the fast builds when possible. The slow builds are slow. Oppositely, most things don't really need to slow builds. We have little to no CI that actually depend on these.

Right, but this is really only true for CI. Our production consumers, release tooling, kubeadm, etc. expect a cross build to be available when traversing a marker.

Email to follow on version marker updates.

BenTheElder commented 4 years ago

it's true for anyone doing CI though, which is sort of the point of CI markers. full blown releases are different.

On Thu, Aug 6, 2020 at 9:48 AM Stephen Augustus notifications@github.com wrote:

Heya, any updates on this since @spiffxp https://github.com/spiffxp's last comment?

Yep. Chatted w/ Aaron and the description of the issue reflects his request now.

Based on chats this week with key Release Team members this remains high-priority. Anything we could delegate here?

There's still a bit of cleanup to do here on my part around the generic version markers.

i think we should have something like:

ci/latest ci/latest-x ci/latest-x.yy release/stable release/stable-x release/stable-x.yy

and that's all that we need...not sure why the project would need anything else?

There are also the release/latest**, which represents pre-release versions.

https://github.com/kubernetes/test-infra/blob/master/docs/kubernetes-versions.md was updated recently to discuss the markers currently in use.

markers with -fast suffix seems fine for k8s maintainers, but for external consumption it does not make much sense.

+100

IMO, we should remove all k8s- markers.

Already part of the plan, described under Generic version markers are not explicit.

skew test jobs for e.g. kubectl could use ci/latest-x.yy explicitly instead of markers that imply N-1, N-2. SIG CLI could upgrade this on each cycle or we could have automation in test-infra to update the jobs.

Agreed. Described in Manually created jobs using generic version markers can be inaccurate and part of Refactor any non-generated jobs using generic version markers.

Most of CI should be consuming quick builds. If I wanted to support CI builds in a tool like kind I'd want the fast builds when possible. The slow builds are slow. Oppositely, most things don't really need to slow builds. We have little to no CI that actually depend on these.

Right, but this is really only true for CI. Our production consumers, release tooling, kubeadm, etc. expect a cross build to be available when traversing a marker.

Email to follow on version marker updates.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/sig-release/issues/850#issuecomment-670046645, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHADK6FGOIRUZB4YQRWBGLR7LNGJANCNFSM4JJGWGLA .

justaugustus commented 4 years ago

Opened https://github.com/kubernetes/community/pull/5268 and https://github.com/kubernetes/test-infra/pull/19686 to move the version markers doc to k/community.

justaugustus commented 3 years ago

/remove-priority critical-urgent /priority important-soon

spiffxp commented 3 years ago

the stableN versions are moved forward after the v1.y.0 release is cut, so that stable1 refers to v1.y.0, the most recent stable release, stable2 refers to v1.y-1.0, the previous stable release, etc.

this hasn't happened by now, and it should have (ref: https://github.com/kubernetes/test-infra/issues/19922#issuecomment-780789250)

spiffxp commented 3 years ago

Markers in k8s-release-dev were rolled forward, markers in kubernetes-release-dev were not. PR to address here: https://github.com/kubernetes/test-infra/pull/20887

fejta-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

fejta-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle rotten

spiffxp commented 3 years ago

/remove-lifecycle rotten We got burned by k8s-beta again as part of https://github.com/kubernetes/kubernetes/issues/103697

I know we're like... right at test freeze. And I haven't paged in all the historical context or job rotation implications etc. But I'm really tempted to say "just slam everything over to latest-1.xy and be done with it". IMO rhe contributor clarity would far outweigh the toil of updating some jobs every N months.

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 3 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 2 years ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

k8s-ci-robot commented 2 years ago

@k8s-triage-robot: Closing this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/850#issuecomment-991940143): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues and PRs according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue or PR with `/reopen` >- Mark this issue or PR as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
justaugustus commented 1 year ago

/reopen My old friend continues in https://github.com/kubernetes/test-infra/pull/28079...

k8s-ci-robot commented 1 year ago

@justaugustus: Reopened this issue.

In response to [this](https://github.com/kubernetes/sig-release/issues/850#issuecomment-1322893840): >/reopen >My old friend continues in https://github.com/kubernetes/test-infra/pull/28079... Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
justaugustus commented 1 year ago

k8s-beta rears it head again in the 1.26 release. See this unmerged PR as an example of fixes: https://github.com/kubernetes/test-infra/pull/26028

@xmudrii is on the right track by opening this issue for k8s-stable4 marker support as a stopgap: https://github.com/kubernetes/sig-release/issues/2094

I'm happy for someone else to attempt closing this, but it may just as well make sense for me to tackle given the context.

justaugustus commented 1 year ago

This plan in the description may be still the right path forward, but take caution and ask questions as it's been a while since I've updated it.

Plan of record

xmudrii commented 1 year ago

@justaugustus -- I'd be happy to help with this! Let me take a look into the current situation and then we can sync about this if needed.

cici37 commented 1 year ago

I would be happy to help with this work. Please let me know if help is needed :)

justaugustus commented 1 year ago

Sounds good. Please work together on this, Cici + Marko! /assign @cici37 @xmudrii /unassign

justaugustus commented 1 year ago

(Linking the k8s-stable4 issue that @xmudrii opened: https://github.com/kubernetes/sig-release/issues/2094)

xmudrii commented 2 months ago

I'm not working on this at the moment. /unassign