Harbor as a cross-cloud global registry for the Kubernetes Project

[hh]

Is your feature request related to a problem? Please describe.

The #k8s-infra-wg exists within the Kubernetes community to support the infrastructure underlying the project.

Currently the Google donated cloud-credits are largely spent on k8s.gcr.io as the primary registry container artefacts are pulled from there. It's on the order of 50-60% of the total spend.

This cost should be distributed by allowing caches or distributed mirror solutions that can be run locally by kubernetes providers / clouds and picked up by users when deploying Kubernetes and pulling the k8s.io released OCI images.

Describe the solution you'd like

As Harbor is also a CNCF project, I think it would be a great solution, creating a cohesive Cloud Native story for registry.k8s.io as a global image/artefact distribution.

Ideally we have a solution that points to registries hosted at Google, Microsoft, Amazon, and a few others. Possibly using their local cloud specific registries, but having a clear option to deploy Harbor in a best practice and community supported manner.

Describe the main design/architecture of your solution

It's an ongoing discussion, which is evolving but some initial ideas:

• Maintain a mapping of Autonomous System Numbers for cloud providers networks that have a local mirror/cache registry.
• Use the SRC ip to redirect (via HTTP/302 or split-horizon DNS) to cloud local mirror
• Dynamically modifying the manifest.json to redirect to cloud local mirror (may have token issues, as most registries do not allow you to pull these blobs without auth)

Describe the development plan you've considered

• Deploy harbor.packet.k8s.io on Packet
• Deploy harbor.microsoft.k8s.io on Azure (or a registry hosted by them)
• Deploy split-horizon DNS PoC : registry-dns.k8s.io
• Deploy http redirect proxy PoC : registry-302.k8s.io
• Show the two PoCs pulling images locally using the two

I have two engineers at ii.coop (@hh and @bobymcbobs) and an evolving team at Microsoft that are keen to help.

Additional context

Some initial conversations in the various slacks:

• https://cloud-native.slack.com/archives/CC1E09J6S/p1615152094159700
• https://kubernetes.slack.com/archives/CCK68P2Q2/p1615174010107500

@bobymcbobs initial explorations for Possible implementations for registry.k8s.io:

• https://hackmd.io/@TKToYPauRJ-u_mNRBOh4HQ/HyJ7yazQO

[hh]

Related: Declarative Deployment of Helm - https://github.com/goharbor/harbor-helm/issues/884

[dims]

cc @justaugustus

[hh]

Lot's of good links over here as well: https://github.com/distribution/distribution/issues/3379#issuecomment-799882488

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.