[Umbrella Issue] Create a Image Promotion process

[dims]

Split from https://github.com/kubernetes/k8s.io/issues/153 (see that for some context)

cc @javier-b-perez @mkumatag @listx

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

[spiffxp]

/remove-lifecycle stale

[spiffxp]

I guess one last thing I'd like to ensure before we call this done is make sure the promoter jobs are running in community-owned infra (re: https://github.com/kubernetes/k8s.io/issues/157#issuecomment-465755277)

[claudiubelu]

I guess one last thing I'd like to ensure before we call this done is make sure the promoter jobs are running in community-owned infra (re: #157 (comment))

Then I think this will be of interest to you.

So, at the moment the Image Promoter for the Kubernetes E2E test images is currently using a few Windows build nodes to build the Windows images, those nodes are in Azure on the CNCF subscription, the same one that is being used to run the sig-windows test jobs.

There was a suggestion to not rely on them anymore, to try to build all the images on the Image Builder node. I have managed to do that with docker buildx and no Windows build nodes, and I've had all the Conformance tests pass with the images built with docker buildx. You can see the proposed PR for this here: [1]. However, there are a few compromises that are to be made for that to be possible. Please read the PR description for more information on that, approval or feedback on that will be greatly appreciated. In any case, this PR depends on this PR [2], which changes the Windows base image from servercore to nanoserver, which is 10x smaller in size. The base image switch was a success, and the nanoserver-based image are currently being used by all the sig-windows test jobs and are passing: [3]

[1] https://github.com/kubernetes/kubernetes/pull/93889 [2] https://github.com/kubernetes/kubernetes/pull/89425 [3] https://testgrid.k8s.io/sig-windows-sac#aks-engine-azure-windows-1909-master

[spiffxp]

From discussion during most recent meeting, items that are preventing us from closing this out:

• Ensure that the image promotion jobs run on community-owned infrastructure vs. google.com-owned infrastructure (https://github.com/kubernetes-sigs/k8s-container-image-promoter/issues/269)
• Ensure that the documentation for the promotion of everything is up to date
  • Document webui access to k8s.gcr.io (https://github.com/kubernetes/k8s.io/issues/1371)
  • Document how to backfill pre-VDF images into post-VDF subdir path (https://github.com/kubernetes/k8s.io/issues/1282)
  • Decide and document rules for what can be hosted as staging projects (https://github.com/kubernetes/k8s.io/issues/915)
  • Document as a known-issue, or fix, the fact that staging registries don't have retention limits (https://github.com/kubernetes/k8s.io/issues/525)
  • Anything else?
• Ensure that the community (someone who isn't @listx or @thockin) is able to respond to an alert, possibly proven via some kind of DiRT (Disaster Recovery Training) scenario

[spiffxp]

/assign @justaugustus @thockin Seems like there was discussion in #wg-k8s-infra about other things to finalize this (from https://kubernetes.slack.com/archives/CCK68P2Q2/p1603988393101900 onward)

Please update this issue accordingly

[spiffxp]

/sig release /area release-eng

[thockin]

@justaugustus Ball is in your court, IMO. I'd love to do a test as CIP exists, make sure the runbook and whatever docs are up to snuff.

On Thu, Oct 29, 2020 at 1:59 PM Aaron Crickenberger < notifications@github.com> wrote:

/sig release /area release-eng

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/kubernetes/k8s.io/issues/157#issuecomment-719018352, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABKWAVERUUFUBT3ZQ6S4363SNHJRZANCNFSM4GIKZDGA .

[justaugustus]

I've been poking around and I just wanted to come back and checklist-ify what @spiffxp mentioned above:

• [x] Ensure that the image promotion jobs run on community-owned infrastructure vs. google.com-owned infrastructure (kubernetes-sigs/k8s-container-image-promoter#269)
• [ ] Ensure that the documentation for the promotion of everything is up to date
  • [x] (https://github.com/kubernetes/k8s.io/pull/1377) Document webui access to k8s.gcr.io (#1371)
  • [ ] Document how to backfill pre-VDF images into post-VDF subdir path (#1282)
  • [ ] Decide and document rules for what can be hosted as staging projects (#915)
  • [ ] Document as a known-issue, or fix, the fact that staging registries don't have retention limits (#525)
  • [ ] Anything else?
• [ ] Transitional - Add senior Release Managers as admins
  • [x] (https://github.com/kubernetes/k8s.io/pull/1392) Enable @kubernetes/sig-release-admins with artifact admin access
  • [ ] Enable non-project owners (artifact admins) to receive auditor alerts (ref: https://github.com/kubernetes/k8s.io/pull/1392#discussion_r517004126, https://kubernetes.slack.com/archives/CJH2GBF7Y/p1604443633126800?thread_ts=1604438283.120800&cid=CJH2GBF7Y)
  • [ ] Ensure that @kubernetes/sig-release-admins are able to respond to an alert, possibly proven via some kind of DiRT (Disaster Recovery Training) scenario

---

Everything below I think is additive and not strictly required to close this issue out.

As I learn more, I'll add a SIG Release tracking issue with additional things we may need to think about like:

• eventually enabling select @kubernetes/release-managers artifact admin access
• review/approval rights over docs pertaining to the process
• maybe some additional doc cleanup/reorganization
• image/file promotion tool consolidation

From @thockin in #wg-k8s-infra:

Hi all! Now that promoter is humming, do we think it would be worthwhile to consider moving the k8s.gcr.io directory to its own repo? Would make it easier to segment email and stuff

cc: @kubernetes/release-engineering

[justaugustus]

Another task that fell out of review in https://github.com/kubernetes/k8s.io/pull/1392:

• Enable non-project owners (artifact admins) to receive auditor alerts (ref: https://github.com/kubernetes/k8s.io/pull/1392#discussion_r517004126, https://kubernetes.slack.com/archives/CJH2GBF7Y/p1604443633126800?thread_ts=1604438283.120800&cid=CJH2GBF7Y)

Added to the checklist above.

[spiffxp]

I'm assuming this remains unfinished given the issues linked in https://github.com/kubernetes/k8s.io/issues/157#issuecomment-721397190 remain open

[spiffxp]

/milestone v1.21

[spiffxp]

/milestone v1.22

[spiffxp]

/milestone v1.23 I'm moving to Blocked as I'm not sure what the status of this is anymore

[spiffxp]

/milestone clear Clearing from milestone, I'm not sure what remains to be done

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[cpanato]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[riaankleinhans]

/remove-lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[ameukam]

I think we are done with this. Image promotion is now part of the release process and use by the different SIGs and subprojects.

Thank you everyone for the work done! /close

[k8s-ci-robot]

@ameukam: Closing this issue.

In response to [this](https://github.com/kubernetes/k8s.io/issues/157#issuecomment-1221106370): >I think we are done with this. Image promotion is now part of the release process and use by the different SIGs and subprojects. > >Thank you everyone for the work done! >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.