BenTheElder
commented
6 months ago There is no real answer to solve this problem. The only solution we got is: Don't restrict internet access for at least one endpoint in your environment.
Current solutions:
- Don't restrict internet access on one host and pull the images via that host
- Use a pull-through cache on a restricted host and unblock the domains as they change, accepting the risk that they can change at anytime
- Use a paid distribution which is sustainably funded by the contract you enter
- Participating in the project's hosting needs in such a way that we can afford to commit to a particular backing host and have the funding and staffing to delay and communicate any changes (remember the entire internet will depend it for free...)
The reasons for this are outlined clearly in the README and previous discussions: We need flexibility on hosting and last time users got the impression that we'd use a specific backing host and not change it that severely limited our ability to keep hosting sustainable.
And again, we're not preventing you from pulling from a restricted endpoint into your mirror based on the hostnames/IPs that you observed but you take on the risk that we may change the backing hosts at any time. Images already mirrored in your pull-through mirror will be available and you'll just have to update the list of backends you permit.
We're not going to document the endpoints, because then people get the impression that they can make demands about how and when that list is updated, which is not something we have the resources to do.
Please understand that the credits and funding that SIG K8s Infra stewards are responsible for hosting the entire project including CI, scale testing, automation, release pipelines, binary and image hosting, etc. for the 100s of repos under the Kubernetes organization.
We nearly ran out of resources on the GCP account that was funding ~all of this (except a little on other vendors at the time and a lot on legacy Google-internal accounts that can only be accessed by Googlers and had a separate budget) because of excessive spend on image hosting which was consuming more than 2/3 of the total spend and way in excess of the $3M annual public GCP budget (in addition to infra on those internal accounts which e.g. hosted binary releases), which was kept online only by injecting another $700,000 while cutting spend on CI and other project-benefitting infrastructure.
We cannot afford to create commitments that are not resourced.
Please note that there is also no SLA with regards to uptime because this is a free service people work on part time, including people who have other primary work or are entirely volunteer, so if you have strict availability guarantees you need to mirror or use a paid distribution regardless of network restrictions.
We're also subject to the network restrictions of the hosts available to us (ref #138 and other issues) and again without alternate resources there's nothing we can do about it. I highly recommend mirroring or using a distribution with a paid SLA for serious production usage.
Unfortunately you'll either have to deal with the limitations and choose a mirroring approach that works for you between the trade-offs in how you choose to depend on the stability of our hosting, unless your organization can make a sufficient long term funding + staffing commitment to give us room to make more unbounded commitments to users.
Is there an existing issue for this?
What did you expect to happen?
Hey, When working in disconnected environments it is normal to have mirror registry somewhere. But this somewhere is normaly also a restricted location which won't allow free unrestricted internet access. As I conduct from multiple closed Issues and your Docs. There is no real answer to solve this problem. The only solution we got is: Don't restrict internet access for at least one endpoint in your environment. This cannot be our allready provided mirror registry. Because we need to secure it. Instead we would need to implement another Registry, outside of our network without any restrictions. And use it as a proxy registry endpoint. The moment this one is not restricted. We could import everything from there. Which in turn makes the whole restriction pointless.
The only two solution I see right now:
I believe doing hacky workarounds to get your images into a secure environment devalues the great service you provide and no sane and liable person should do this. Also having a "pull from everywhere" mirror registry does not seem to be a secure choice either.
I was hesitant to even bring up the topic, since your service is not responsible for our needs. In the end I decided to start this discussion again, since alot of kubernetes-sigs images are officially and foremost on this registry. With the high impact of Kubernetes and the neccessity of trusting images and image registries I cannot be the only one who needs a trusted endpoint for this registry. You mentioned here participating Vendors sometimes offer your Repositories in their Licence model. Which is not the case for the images we need.
Maybe I am missing something! I would love to hear feedback and suggestions, that are feasable for secure and disconnected environments that do not require to create a man-in-the-middle registry or carry important images with an USB stick to their destination.
Kind regards.
Debugging Information
No technical Problem. So no Debugging
Anything else?
No response
Code of Conduct