Develop a prow migration plan

[spiffxp]

Break out into issues when we've gotten consensus

Current sketch is:

• decide whether we want a build cluster, and/or how many we we want
• allow prow.k8s.io to schedule jobs on our build cluster(s)
• setup a prow instance (in aaa?)
• figure out how we want to migrate/cut-over github traffic

Some concerns include:

• jobs that run in same cluster as prow.k8s.io to take advantage of the ghproxy instance that lives there
• using a new github bot user/token so as not to conflict with k8s-ci-robot while prow.k8s.io continues to use it
• setting up boskos in our build cluster(s)
• setting up greenhouse in our build cluster(s)
• setting up projects for boskos to manage
• migration of google-owned buckets like kubernetes-jenkins
• on-call

[spiffxp]

/wg k8s-infra /sig testing /area cluster-infra /assign

[spiffxp]

I prototyped a single build cluster here https://github.com/kubernetes/k8s.io/pull/806

I have it hooked up to my personal prow instance here: https://prow.bashfire.dev

[spiffxp]

Trying to work through what the plan looks like, it's unlikely we'll get to discussing the actual "migrate prow.k8s.io" part in much detail this week.

https://docs.google.com/document/d/1g7RBDhbTAH1VBMl2kJZR71OfzM6cJF277zbysd2AoCQ/edit#

[spiffxp]

Discussed during today's meeting, came away with:

• We'll do two build clusters, each in their own project (trusted, untrusted)
• Untrusted build cluster will use same machine type as used by prow.k8s.io's existing build cluster (n1-highmem-8)
• Will use regional clusters
• Will start with e2e project pool that is not sig/job-specific
  • Adjust when costs get to unknown/unexplained size
  • Consider per-sig project pools
• Naming Is Hard:
  • k8s-infra-prow-build - untrusted build cluster (and project, serviceaccount, etc)
  • k8s-infra-prow-build-trusted - trusted build cluster (and project, serviceaccount, etc)
  • k8s-infra-e2e-{suffix} - e2e projects intended to be pinned to a job
  • k8s-infra-e2e-boskos-{nnn} - e2e projects intended to be managed by boskos
• What I'm aiming for in two weeks:
  • clusters hooked up to prow.k8s.io
  • confirmed migration of:
  • ci-kubernetes-node-kubelet
  • ci-kubernetes-e2e-gci-gce
  • (an image pushing job?)
  • dev/test cycle for migrating jobs

[spiffxp]

https://github.com/kubernetes/k8s.io/pull/830 implements all of the above except dev/test cycle

[spiffxp]

Opened up some more issues to represent followup work:

• [x] - https://github.com/kubernetes/k8s.io/issues/841 - migrate release-master-blocking jobs to prow-build cluster
• [x] - https://github.com/kubernetes/k8s.io/issues/842 - setup greenhouse
• [ ] - https://github.com/kubernetes/k8s.io/issues/843 - setup ghproxy
• [x] - https://github.com/kubernetes/k8s.io/issues/844 - better ACLs to empower more people
• [x] - https://github.com/kubernetes/k8s.io/issues/845 - postsubmit jobs to deploy cluster resources
• [x] - https://github.com/kubernetes/k8s.io/issues/846 - migrate jobs away from gs://kubernetes-release-dev

[MHBauer]

Learning about sig-node ci, can I help with or shadow along changes to ci-kubernetes-node-kubelet?

[spiffxp]

@MHBauer add yourself to the k8s-infra-prow-viewers@kubernetes.io group in groups/groups.yaml and let me know if you can see https://console.cloud.google.com/monitoring/dashboards?project=k8s-infra-prow-build

[helenfeng737]

I can help with the migration work too if needed. Already added myself in k8s-infra-prow-viewers

[spiffxp]

About ready to enumerate:

• setting up a staging prow instance to experiment with running in community infra
• how to shift traffic from prow.k8s.io

[spiffxp]

/assign @ameukam since also working on staging prow (ref: https://github.com/kubernetes/k8s.io/issues/1394)

[spiffxp]

/priority important-longterm I don't know how much time I'll have to work on completing a plan by end of v1.21

[fejta-bot]

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community. /lifecycle stale

[ameukam]

/remove-lifecycle stale

[spiffxp]

/lifecycle frozen

[spiffxp]

/milestone v1.23

[spiffxp]

/remove-priority important-longterm /priority important-soon

[spiffxp]

The proposal linked in this issue likely needs to be refreshed, but dropping a few thoughts before I forget.

Blockers to migrating all jobs over to a community-run build cluster include:

• setting resource requests/limits on all jobs to properly trigger autoscaling
• using community-owned projects or granting privileges to the community-owned build cluster and boskos to manage google.com projects
  • manually pinned projects (e.g. https://github.com/kubernetes/test-infra/issues/23822, the list from https://github.com/kubernetes/k8s.io/issues/1469)
  • gke projects (https://github.com/kubernetes/test-infra/issues/23778)
• more budget than we have capacity for based on our current spend w/ scalability jobs migrated over

Blockers to shutting down prow.k8s.io and then standing it all up over in kubernetes-public include:

• all of the existing kubernetes/test-infra tooling / manifests assume deployment to its own cluster vs. a shared cluster
• allowing a community-owned prow to trigger builds in the google.com-owned "default" build cluster (if we fail to migrate all jobs over
• allowing a community-owned prow to write/update gs://k8s-testgrid
• allowing a community-owned prow to write/update gs://kubernetes-jenkins
• allowing a community-owned prow to write to the google.com-owned k8s-prow project
• prow.k8s.io currently runs jobs for more than the kubernetes community (https://github.com/kubernetes/test-infra/issues/12863)
• (not sure what else I'm missing here...)

[ameukam]

/milestone v1.24

[ameukam]

/milestone v1.30 /lifecycle clear

Rebooted this effort. See: https://docs.google.com/document/d/1erBhuCwY26d0UfPbzt8lEj6bYT2hOUKzc2j36YHVqfM/edit?usp=sharing

[ameukam]

/milestone v1.31

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[ameukam]

/remove-lifecycle stale

We have a timeline for this and an overall plan to do the migration. Technical requirements need to be discussed.

[ameukam]

We have a migration plan established: https://docs.google.com/document/d/1erBhuCwY26d0UfPbzt8lEj6bYT2hOUKzc2j36YHVqfM/edit

Execution is planned: https://groups.google.com/a/kubernetes.io/g/dev/c/qzNYpcN5la4

cc @BenTheElder @michelle192837

/close

[k8s-ci-robot]

@ameukam: Closing this issue.

In response to [this](https://github.com/kubernetes/k8s.io/issues/752#issuecomment-2301386954): >We have a migration plan established: https://docs.google.com/document/d/1erBhuCwY26d0UfPbzt8lEj6bYT2hOUKzc2j36YHVqfM/edit > >Execution is planned: https://groups.google.com/a/kubernetes.io/g/dev/c/qzNYpcN5la4 > >cc @BenTheElder @michelle192837 > >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes-sigs/prow](https://github.com/kubernetes-sigs/prow/issues/new?title=Prow%20issue:) repository.