search

kubeflow / pipelines

Machine Learning Pipelines for Kubeflow
https://www.kubeflow.org/docs/components/pipelines/
Apache License 2.0
3.6k stars 1.62k forks source link

Move away from Google Container registry, and cloud build system for image builds #11208

Closed HumairAK closed 1 month ago

HumairAK commented 1 month ago

As per the last KFP call on 2024-09-11

It was decided that the following pieces that are part of the release process are still highly dependent on google infra:

  • Buildling mechanism, currently we’re using cloud build mechanism
  • container registry, technically you can use the github container registry, but not sure if there’s a ratelimiting

Suggestion:

This looks like we can easily resolve this dependency by utilizing Github Actions as the build mechanism, and using Github Container Registry. With KF as a CNCF project we should have enterprise access to GitHub Container Registry.

HumairAK commented 1 month ago

cc @zijianjoy / @chensun

HumairAK commented 1 month ago

@rimolive will contact CNCF and follow up

rimolive commented 1 month ago

/assign @rimolive

HumairAK commented 1 month ago

/assign @HumairAK

HumairAK commented 1 month ago

cc @chensun / @zijianjoy I've added a PR for a workflow that will build and push images to the kubeflow/pipelines github container registry

how does this sound to you? based on their pricing it seems it's free for public repos: https://github.com/pricing

image

If there's anything missing that we need from enterprise, we can reach out to CNCF for enterprise access (however that works)

HumairAK commented 1 month ago

I've also started some slack discussions in the CNCF kubeflow slack channels here and here to discuss ghcr as a candidate

Edit: Currently it seems there is no consensus on the standard image registry for KF to use, some are using docker hub, and some are using GHCR. My suggestion is that we go with ghcr for now as the integration is a bit easier, the github workflow already has built in credentials via $GITHUB_TOKEN, and their pull rate limit seems a lot more generous. We may need to revisit once there's a finalized location for all of KF to host its images.