Move to GCP Prow CI instead of Kubernetes's Prow CI

[BenTheElder]

xref: https://github.com/kubernetes/test-infra/issues/12863

It would be great if we could move this project to use https://github.com/GoogleCloudPlatform/oss-test-infra instead of https://github.com/kubernetes/test-infra since it is a Google project and not part of the Kubernetes project.

In the long term, we are migrating Kubernetes's CI to community managed infrastructure and those resources are allocated for the Kubernetes project.

A few projects have moved already, as you can see in https://github.com/kubernetes/test-infra/issues/12863

For googlers: see go/oss-prow-onboard

[BenTheElder]

cc @dims @bobbypage https://github.com/kubernetes/test-infra/issues/12863#issuecomment-1526126716

[bobbypage]

Thanks for bringing this up.

Currently, cAdvisor is using a mix of github actions and prow. The main thing we use prow for is for running the integration tests.

There are two jobs defined in prow:

1. pull-cadvisor-e2e
2. ci-cadvisor-e2e

build/prow_e2e.sh is invoked by the node e2e runner, followed by build/integration.sh, see here.

The github action test-integration currently runs make docker-test-integration, which will end up running build/integration-in-docker.sh which runs the same integration tests but inside a dockerized container.

As a result, if we need to drop prow support, I think it may be simpler to simply drop all the prow jobs and only rely on github actions, since we are already running the integration tests there. I don't see any reason we need to depend on prow.

Thoughts @dims @iwankgb @Creatone?

[iwankgb]

I think that using GitHub Actions is a good idea. Everything else will be more or less Google-specific and more difficult to grasp for non-Googlers. I'm not sure if we can use free runners in long run, as they can stop being free or their limitations (e.g. number of parallel builds) may start biting us.

[BenTheElder]

As a result, if we need to drop prow support, I think it may be simpler to simply drop all the prow jobs and only rely on github actions, since we are already running the integration tests there. I don't see any reason we need to depend on prow.

To be clear: You don't need to drop prow, but if you want to keep using prow please move to the instance Google provides for Google projects instead of the original instance Google funds through the CNCF for Kubernetes which now also has funding from AWS and will eventually be entirely dedicated to Kubernetes projects.

Multiple other projects have migrated, it's not terribly involved, we just need to stop conflating funding and operating these. The Google prow also operates on a model where teams provide (and pay for) the execution cluster while the Prow team operates the central controllers etc.

That said if you want to drop it instead, that's fine.

[BenTheElder]

@dims WDYT? This will be the last remaining project aside from apisnoop (moving to k8s org) and containerd (at least CNCF...) after https://github.com/kubernetes/test-infra/pull/32089

xref: https://groups.google.com/a/kubernetes.io/g/dev/c/p6PAML90ZOU

[BenTheElder]

I'm not sure if we can use free runners in long run, as they can stop being free or their limitations (e.g. number of parallel builds) may start biting us.

YMMV. So far this doesn't seem to be a huge issue though it's totally possible that they become non-free.

In that case, https://github.com/googlecloudplatform/oss-test-infra#oss-prow does exist which Google can fund/operate.

I have mixed feelings on this, cadvisor and k8s have history, even if I'd prefer to only have Kubernetes or at least CNCF projects on the Kubernetes project resources just to have clear policy ...

[dims]

@BenTheElder i'd vote us to keep cadvisor and cri-o/containerd as exceptions as they are integral to what we do in sig-node/kubelet. At least until we can get rid of cadvisor for good for metrics (not sure when!)

[BenTheElder]

There's a new wrinkle: https://groups.google.com/a/kubernetes.io/g/dev/c/p6PAML90ZOU

cadvisor jobs are impacted, and they probably missed the notification because it's outside of the Kubernetes org but using our CI.

[iwankgb]

@bobbypage do we really nead Prow at all? Same tests are run on GH Actions anyway.

[dims]

@iwankgb i already took care of the cadvisor jobs - https://github.com/kubernetes/test-infra/pull/32810