Add support for downloading from Rapid without streamer

[p2004a]

This PR enables downloading Rapid packages without usage of streamer.cgi server side component and overriding the repo from which packages are downloaded. Rapid package is being downloaded by fetching each file from the pool folder individually. The files are not pulled sequentially but in parallel leveraging HTTP2 multiplexing.

The repo and parameters of downloading are being controlled by PRD_* environment variables. They are being used for this purpose instead of flags because they are by default inherited by all the processes in the process tree which allows to override behavior of pr-downloader globally impacting invocation of the spring-launcher, new lobby, chobby which helps with experimentation without a need to patch arguments via multiple layers of invocations. See changes in main.cpp for the list of new variables and what they control.

Implementation

Because of the state of the code, next to introducing this feature, most of the HTTPDownloader.cpp had to be rewritten entirely dropping legacy no longer used codepaths that didn't work at all anyway. The new implementation additionally retries transient server errors and correctly verifies file hashes before saving the files.

This PR also combines a bunch of small additional changes and fixes for issues I've noticed during implementation and got into the way, e.g:

• Wrong naming of int serialization functions
• Streamer assuming happy path when using sendfile
• Not failing execution when Sdp is corrupted
• Removes a bunch of dead code (not all, more PRs with refactoring will come, don't want to make this PR even bigger)
• Add support for overriding springfiles URL

Testing

To verify behavior of the downloader, I've implemented python functional tests that perform blackbox testing of the pr-downloader functionality, comparing streamer and non-streamer behavior. They are written and licensed so that if other more sane and concise implementations like sprd should be able to easily import them and use for testing too.

I've also added instructions for setting up container with testing repository I've used to compare streamer and non-streamer behavior under different network conditions. Overall I'm happy with the performance, there aren't situations I'm aware of where streamer is more than 2x quicker then non-streamer implementation, and the slower the connections, the more similar they are.

For the most realistic test I've set up a hosting using CDN and imported one of the latest versions byar:test packages to test downloading in potentially real setup. You can try it out yourself quickly as I've compiled and published this version of pr-downloader into container image:

time docker run -v /tmp:/tmp --rm \
  -e PRD_RAPID_USE_STREAMER=false \
  -e PRD_RAPID_REPO_MASTER=https://bar-rapid.p2004a.com/repos.gz \
  -e PRD_MAX_HTTP_REQS_PER_SEC=0 \
  us-central1-docker.pkg.dev/playground-338021/p2004a-images-repo/prd:latest \
  --filesystem-writepath /tmp/pr-downloader-test --rapid-download byar:test

~Don't go beyond 600req/s or you will probably get DDoS blocked~ (EDIT: got the limit lifted with the help from support). For me, as I'm quite close to the repos.springrts.com (<10ms) and with fast connection it take ~22s to download from repos.springrts.com via streamer, and ~26s~21s to download from the CDN without streamer which is very good.

Caveats

• Sdp index file contains only uncompressed file sizes, so this implementation currently doesn't show correct compressed real download size in the progress bar, but the uncompressed size. Happy to hear suggestions for ideas. We could add a second file with real gzipped file sizes if that's important...
• After this gets merged, it might not work well as I'm not sure if curl in spring-static-libs is compiled with HTTP2 support (to be checked)

[Beherith]

Hi dude, regarding Allow overriding url of rapid repo master

I dont quite know how that bit works, just that prd has 2 use cases, one where it is linked into engine, and can download stuff from ingame, and one external, static version which is used by the updater, and both use cases will probably need alternate repos gz urls :)

[p2004a]

It's fully flexible, we can have launcher config where you set them differently. But why would game and launcher need separate repo masters?

[Beherith]

"It's fully flexible, we can have launcher config where you set them differently. But why would game and launcher need separate repo masters?"

So launcher could look at the master, and once people go ingame, the lobby server could direct them to more location-appropriate mirrors.

[p2004a]

@Beherith @gajop I believe this should be ready to review now, see the top description for summary of changes.

Re:

So launcher could look at the master, and once people go ingame, the lobby server could direct them to more location-appropriate mirrors.

For now, I'm trying to use a single CDN that distributes files to a bunch of Pops around the world. If there is desire to set different mirrors from lobby, it shouldn't be hard to make this functionality available from lua to be used by chobby. AFAIK will use pr-downloader as a binary so that's covered.

[p2004a]

FYI: https://bar-rapid.p2004a.com/repos.gz contains now full byar and byar-chobby repos that are live mirrored (sync every 7mins) from the https://repos.springrts.com/.

[Beherith]

Oh man did you ever make my week dude, i cant wait for the weekend to test this :D

[Beherith]

Should i squash this?

[p2004a]

Should i squash this?

I try to make the commits pretty self contained and tested at each point (not random "more code", "fixed stuff") so having them in history for anyone doing git blame in the future might be better idea then squashing.

[Beherith]

Ok, I wont squash, it all seems lovely. Just one question, will migration to this prevent downloading versions older than the changeover point?

[p2004a]

CDN has full repository of the game, full history, so lobby will be able to fetch old version of the game for any replay etc. (It's ~60000 files, ~5GiB)

Also to be 100% clear: this PR only adds code paths that can be selected using environment variables to be able to fetch from CDN etc, and I've prepared the CDN itself. By default pr-downloader is still going to use https://repos.springrts.com/ using the streamer.cgi download method.

[Beherith]

Awesome! One more silly question: can the CDN url be changed while launcher is running? (Lets say that once the person logs in, we see that we have a CDN in his region, and want to serve him from there)

[p2004a]

From the launcher: no problem. From the lobby: depends if the lobby is using pr-downloader as library or as executable, but I believe as a library so not with current implementation. I would need to add API for that.

But that shouldn't be something that we will need or even want to do. The CDN is globally distributed, there shouldn't be a situation where not using CDN is faster in any noticable way (To roll this out to the users, I will make sure that there is option in launcher to opt out to the old method is something is broken for somebody with the new method)

Currently to bound the cost I've configured CDN to use the "Volume tier" that uses 8 data centers (4xUS, 3xEurope, Singapore), but if that's not considered enough it's possible to throw more money at the problem and use ~98 datacenters, pick other CDN etc (Here is page about network of CDN I've picked: https://bunny.net/network)

[p2004a]

Update: I've prepared 3 PRs visible above so that it compiles and works correctly with HTTP2 support on the linux-64 platform when build as part of the spring engine release using docker-build scripts.

I still need to prepare PRs to also make sure that it compiles and works on windows-64 platform in the docker-build.

[p2004a]

I've prepared PR to other repo to make sure that it compiles under docker build and works correctly also on windows-64.

I've also tested a bit more on Windows, fixed one bug and added a bit more diagnostics about download.

The main issue on Windows that I don't know currently how to, and don't plan to try tackle is very slow file IO. The download speed when files are not being created is very comparable to Linux, only a little bit slower, but once the rapid pool files are actually being saved to the disk it's extremaly slow in comparison to the performance I get on Linux. Because of slow file IO the download time jumps from <30s to mulitple minutes on Windows for me.

[p2004a]

FYI: published my rapid syncer code I use for mirroring: https://github.com/p2004a/spring-rapid-syncer

[Beherith]

Looks great to me, will merge after successful engine deployment.