Strategies to handle Linux package download limits

[ABCMoNa]

I'm suggesting to use this issue to identify strategies to counter the recurring issue of 3rd party hosted Linux package repositories hitting download volume limits (#4034).

I can currently think of the following approaches:

• Run own mirror(s) and repository management software (@aethernet mentioned "CloudFlare R2 + workers" )
  • pro: could fully solve issue
  • contra: (probably) undesirable maintenance overhead
  • contra: also depends on finding a sponsor
• Use better compression algorithm for packaging (e.g. zstd for packages, .gz for Debian Release files)
  • pro: If / where possible, reduces download volume for each download; may not be possible for all Linux package repositories / distros; should work for Debian / Ubuntu (but zstd may not work with older distro versions!)
  • contra: requires further analysis, possibly different repository management tools
• Remove older packages (e.g. for Ubuntu, several package versions incl. very outdated ones are available)
  • pro: probably easily achieved
  • contra: may not impact download volume much, since those outdated packages will be downloaded much less often

What do think about those suggestions? Which other strategies seem worthwhile?

Are download statistics available / can they be made available? (Ideally for monthly download volume per location / file / distro + version)?

Thanks! (Feel free to remove this issue if it does not seem to contribute to your fact finding.)

[dmaes]

Suggestion: OpenSUSE's OBS (build.opensuse.org) supports both RPM and DEB, and is free as far as I can see (https://en.opensuse.org/openSUSE:Build_Service_Terms_of_Service#K._Payment)

[aethernet]

Hello,

Run own mirror(s) and repository management software (@aethernet mentioned "CloudFlare R2 + workers" )

As Balena already has a CF enterprise account with unlimited bandwidth.

There's indeed a maintenance cost but it feels reasonable. Another - may be simpler - option would be CloudFlare Pages instead of R2 + Workers. I saw guides on how to host on GitHub Pages and they're similar in terms of functionalities.

To be manageable on our side it would need to be automated as part of the CI. It needs to be a FlowZone step so it runs on a GH worker, which takes the packages that already exist, signs them, creates the repositories metadata files then uploads everything to CF Pages.