search

flatpak / flat-manager

Manager for flatpak repositories
Apache License 2.0
178 stars 43 forks source link

[Feature Request] use rsync to host flatpak repository #48

Open willyywt opened 3 years ago

willyywt commented 3 years ago

A link to https://github.com/flatpak/flatpak/issues/4383

I am a user from China and I am affected by https://github.com/flathub/flathub/issues/813, as GFW in China completely censored Fastly CDN. Currently flatpak repo are only intended to used by a CDN, but it bascially restricted the mirroring method to becoming a reverse proxy, which is proved very toxic to mirror sites: for tuna and sjtug

Rsync rationale

I don't know flatpak very well, but I think flatpak is a repo manager for an ostree repo, which itself seems to be easy for rsync to mirror: https://github.com/ostreedev/ostree-releng-scripts/blob/master/rsync-repos. I suggest flatpak should support an additional remote storage backend using rsync in parallel with the current CDN-like approach.

Rsync advantages

  • Commonly accepted protocol for mirroring
  • Incremental updates
  • Packed transfering of files
  • Safe resume from failed sync
  • Support both unauthorized client and authorized client

Rsync problems

  • Pressure on server resource
  • Still more costly than plain deb/rpm packages (but better than making large number of small http requests)

CDN advantage

  • Doesn't require a strong server
  • Approachable to many (but not more than mirror sites since CDN providers are not approachable everywhere) areas using a relatively low price

CDN-only problems

  • Effectively limits mirror methods to reverse proxy only, which is unacceptable for some mirror sites and irritating for the rest
  • Too many delta file increases the risk of failure because of increased number of http request round trip
  • No safe and incremental way of recovering from failed download

These are not flatpak-specific issues; many FOSS projects are affected by them.

Many mirror sites can support rsync in addition to plain http; for these sites rsync are a proved working way of mirroring and are way easier to manage. If remotes like flathub doesn't have much server resource, they can limit the bandwidth to unauthorized clients and reserve more resource to mirror sites.

TCB13 commented 3 months ago

Maybe the flat-manager community can help a but there because this is the kind of BS the flathub people are trying to pull:

Rsync-based mirroring isn't realistic given our ostree repository contains over 3TB of reachable objects. I'm 100% sure that if e.g. STJUG had to mirror 3TB, that would easily prove more "toxic" as you put it.