Migrate merge-blocking jobs to dedicated cluster: pull-kubernetes-verify

[spiffxp]

What should be cleaned up or changed:

This is part of #18550

To properly monitor the outcome of this, you should be a member of k8s-infra-prow-viewers@kubernetes.io. PR yourself into https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L603-L628 if you're not a member.

Migrate pull-kubernetes-verify to k8s-infra-prow-build by adding a cluster: k8s-infra-prow-build field to the job:

• Make sure any release-branch variants are also updated
• Reference this issue, but do not "fix" it; instead leave this open to monitor the behavior of the job before/after
• Here's an example PR that did this for pull-kubernetes-integration: https://github.com/kubernetes/test-infra/pull/18814

Once the PR has merged, note the date/time it merged. This will allow you to compare before/after behavior.

Things to watch for the job

• https://prow.k8s.io/?job=pull-kubernetes-verify
  • does the job start failing more often?
  • does the job start going into error state?
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-verify&graph-metrics=test-duration-minutes
  • does the job duration look worse than before? spikier than before?
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-verify
  • do more failures show up than before?
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-verify
  • (can be used to answer some of the same questions as above)
• metrics explorer: CPU limit utilization for pull-kubernetes-verify for 6h
  • is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• metrics explorer: Memory limit utilization for pull-kubernetes-verify for 6h
  • is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)

Things to watch for the build cluster

• prow-build dashboard 1w
  • is the build cluster scaling as needed? (e.g. maybe it can't scale because we've hit some kind of quota)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• prowjobs-experiment 1w
  • (shows resource consumption of all job runs, pretty noisy but putting this here for completeness)

Keep this open for at least 24h of weekday PR traffic. If everything continues to look good, then this can be closed.

/wg k8s-infra /sig testing /area jobs /help

[helenfeng737]

/assign

[spiffxp]

/remove-help since @ZhiFeng1993 has it

[spiffxp]

/close Calling this done, thanks for your help!

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to [this](https://github.com/kubernetes/test-infra/issues/18849#issuecomment-683194632): >/close >Calling this done, thanks for your help! Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.