Migrate merge-blocking jobs to dedicated cluster: pull-kubernetes-e2e-gce-network-proxy-http-connect

[spiffxp]

What should be cleaned up or changed:

This is part of #18550

To properly monitor the outcome of this, you should be a member of k8s-infra-prow-viewers@kubernetes.io. PR yourself into https://github.com/kubernetes/k8s.io/blob/master/groups/groups.yaml#L603-L628 if you're not a member.

Migrate pull-kubernetes-e2e-gce-network-proxy-http-connect to k8s-infra-prow-build by adding a cluster: k8s-infra-prow-build field to the job:

• Make sure any release-branch variants are also updated
• Reference this issue, but do not "fix" it; instead leave this open to monitor the behavior of the job before/after
• Here's an example PR that did this for pull-kubernetes-integration: https://github.com/kubernetes/test-infra/pull/18814

NOTE: migrating this job is not as straightforward as some of the other #18550 issues, because:

• it doesn't run for every single PR
• so in terms of appropriate amount of PR traffic, either manually trigger this job, or look for open PRs that have triggered this job (e.g. using the job-history link below), and explicitly /test <this-job-name> on them

Once the PR has merged, note the date/time it merged. This will allow you to compare before/after behavior.

Things to watch for the job

• https://prow.k8s.io/?job=pull-kubernetes-e2e-gce-network-proxy-http-connect
  • does the job start failing more often?
  • does the job start going into error state?
• https://testgrid.k8s.io/presubmits-kubernetes-blocking#pull-kubernetes-e2e-gce-network-proxy-http-connect&graph-metrics=test-duration-minutes
  • does the job duration look worse than before? spikier than before?
• https://storage.googleapis.com/k8s-gubernator/triage/index.html?pr=1&job=pull-kubernetes-e2e-gce-network-proxy-http-connect
  • do more failures show up than before?
• https://prow.k8s.io/job-history/gs/kubernetes-jenkins/pr-logs/directory/pull-kubernetes-e2e-gce-network-proxy-http-connect
  • (can be used to answer some of the same questions as above)
• metrics explorer: CPU limit utilization for pull-kubernetes-e2e-gce-network-proxy-http-connect for 6h
  • is the job wildly underutilizing its CPU limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• metrics explorer: Memory limit utilization for pull-kubernetes-e2e-gce-network-proxy-http-connect for 6h
  • is the job wildly underutilizing its memory limit? if so, perhaps tune down (if uncertain, post evidence in this issue and ask)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)

Things to watch for the build cluster

• prow-build dashboard 1w
  • is the build cluster scaling as needed? (e.g. maybe it can't scale because we've hit some kind of quota)
  • (it will probably be helpful to look at different time resolutions like 1h, 6h, 1d, 1w)
• prowjobs-experiment 1w
  • (shows resource consumption of all job runs, pretty noisy but putting this here for completeness)
• https://monitoring.prow.k8s.io/d/wSrfvNxWz/boskos-resource-usage?orgId=1
  • are there still plenty of free projects in the "GCE project (k8s-infra) usage status" graph?

Keep this open for at least 24h of weekday PR traffic. If everything continues to look good, then this can be closed.

/wg k8s-infra /sig testing /area jobs /help

[RobertKielty]

/assign /remove-help

[RobertKielty]

@spiffxp I submitted a PR for this a few days ago but failed to add "fixes" in order to link it back to this issue.

When you get a chance can you take a look and approve the change? Many thanks.

[spiffxp]

https://github.com/kubernetes/test-infra/pull/19153 has a merge conflict

I'll trigger a few more runs and keep an eye on https://prow.k8s.io/pr-history/?org=kubernetes&repo=kubernetes&pr=94127

This job runs so infrequently that as long as its behavior looks close to the gce or ubuntu e2e jobs I'll be happy with it once the real job is replaced with the changes in the canary (minus the testgrid dashboard change)