search

containerbase / base

The containerbase project's base image source
MIT License
35 stars 40 forks source link

Tools mirror/ caching CDN #972

Open rarkins opened 1 year ago

rarkins commented 1 year ago

Today's approach has some downsides:

  1. Image builds sometime fail for periods if one of the hosts we download from is flaking
  2. Self-hosted users who build their own image may not be allowed or find it easy to "open up ports" to so many hosts

Example of flakiness:

 > [test 13/17] RUN install-tool rust 1.69.0:
#42 0.214 installing v2 tool rust v1.69.0
#42 0.842 curl: (56) OpenSSL SSL_read: Connection reset by peer, errno 104
#42 0.845 Download failed: https://static.rust-lang.org/dist/rust-1.69.0-x86_64-unknown-linux-gnu.tar.gz

We should aim to have centralization and control of such dependencies where possible or necessary. I can think of two main possibilities:

  1. We mirror everything onto github.com, probably as "Releases" because github doesn't yet support "generic" packages in its registry, or
  2. We mirror everything statically onto e.g. S3 and essentially host a public mirror ourselves (paid for by Mend)

Obviously either way we still can have flakiness if our host (GitHub or CDN) has flakiness, but at least it's centralized flakiness and likely less than we get today

viceice commented 1 year ago

related:

viceice commented 1 year ago

maven issues caused by ban^1

rarkins commented 1 year ago

So our goal here is:

If we can achieve this with reasonable cost efficiency, we could then default Containerbase to only use this host instead of the variety we have today.

rarkins commented 1 year ago

Cloudflare's cache reserve is one approach to reducing requests to the origin server(s) and providing more resiliency for those which might have availability problems: https://developers.cloudflare.com/cache/advanced-configuration/cache-reserve/

viceice commented 1 year ago 
          I've registered containerbase.dev just now and propose we aim to use cdn.containerbase.dev

Originally posted by @rarkins in https://github.com/containerbase/base/issues/1507#issuecomment-1733586167

rarkins commented 5 months ago

Update: cdn.containerbase.dev is now ready

I would like to make this opt-in via CONTAINERBASE_CDN=https://cdn.containerbase.dev

Setting this would result in all known hosts to be redirected without the need for the http from/to being all set individually

zharinov commented 5 months ago

Just to clarify: setting CONTAINERBASE_CDN=https://cdn.containerbase.dev/ should transform requests like https://example.com/foo/bar to the form like https://cdn.containerbase.dev/example.com/foo/bar

rarkins commented 5 months ago

I was thinking that it would need to have an explicit list but I suppose you could add a general http rewrite rule? Might break if there's 30x responses though

viceice commented 5 months ago