[Engine] FastDL overhaul

[DBotThePony]

As of 2023, Issues below are still relevant, but solutions are not, please refer to my new comment below.

So, if no one decide to write this feature request, i will do it by myself.

Currently, we got very rough and incompetent FastDL system avaliable (for downloading files from HTTP(S) server). Sure, using HTTP web server is good, and it really improve speed, but only when compared to internal gameserver download mechanism, but, however, our download speed is still low (even on LAN connection!) Why is it currently so bad? There are several reasons:

1. "Fake" latency. When client download a bunch of small files one-by-one it decompress each when it finishes download, and it takes some time and is not really good, since requests are not "piped" to web server. This sometimes make huge latency to fastdl web server even bigger and it is all very sad.
2. Visual: we are not telling user how much we downloaded and how much there are files left, only drawing ambiguous progress bar which make no sense almost all the time.
3. There is no priority control (by gameserver owners), which files to preffer - bzip2 or uncompressed. Since (almost all) server owners are unaware that find . -type f -exec bzip2 -v {} + exists, on most game servers we get only maps compressed, this creates extra latency because client request bzipped files fist (which return 404 by web server), and then it starts to actually download files.
4. And, the most important, multiple download connections. Currently, we download files in a single, blocking thread. How uneffective! Sure, at moment of 2000s, single connection is indeed filling up the entire internet channel of user. But in modern world, download speed is mostly based on how much connections we open to the web server. So, if we open a single connection, we can download files one-by-one at average at 80 kb/s (hell), including the fact that we are on high (>90 ms is already high latency) latency, we are truly saying "No" to modern connection speed capabilities. What if we have high latency to the web server, but at same time we have wide connection channel to that web server? Answer is plain and simple - to fully use potential of user's connection speed to target server we need to open several separated connections to that web server and download different files simultaneously. If you could implement that, that would be a huge step forward to fast, or even very fast first join times. From my own measurments, i could get 1300 kb/s download speed from my web server to a user in Canada using 16 download channels instead of one. And when the user in question use only one channel, his download speed is rough about only 80 kb/s. Isn't it great that we significantly improved download speed of client just by increasing download connections?

So what's the deal with all these paragraphs? In my design we could do next things:

1. We make sure that download/temp exists and it doesn't contain any (bzip2) files
2. We connect to the server as usual. We send "hello" to the server, then get "hello" from the server with sv_downloadables table, and sv_downloadurl, sv_downloadurl_priority (0 = preffer bzip2 first, 1 = preffer uncompressed first), sv_downloadurl_connections (maximal amount of connections to web server). My suggested range for that setting is 1 to 16. Values over 16 or less than 1 should not take any effect on client, default suggested value should be 4 in my opinion
3. Then, we split sv_downloadables into sv_downloadurl_connections parts and create threads/green threads/libcurl/curl.exe instances (curl is preferred since we don't need to deal with newer https, http/2 by yourself) to download files.
4. We download all files simultaneously into download/temp if file is compressed or into download/ directly if file is not compressed using our instances and displaying user status of downloading either by sv_downloadurl_connections bars or as single bar but with explained info on it's header (instead of plain "Downloading...").
5. Unpack downloaded files inside download/temp onto download/. If we got no compressed files (?), then we skip this action. As we decompress files, we remove their .bz2 variant. At this moment, we can display user "Unpacking downloaded files".
6. We are ready to go!

Q: Why do we need sv_downloadurl_connections? A: Because im pretty sure some of server hosters would want to not overload their FastDL web servers (or, even tell server's owner that he can increase this value to whatever he would like with explanation).

This request is really my dream since 2013, and im not sure if this dream would come true or not. But if this would come true, all server owners will have much more abilities to create truly creative content and servers, since we would get much more reliability on builtin download system for variety of servers.

[DBotThePony]

Also there can be "Upvotes/Downvotes" download model. As we are going on downloading files, we can conclude whenever to or not to increase download channels.

"Upvotes" are:

• TTFB is equal/larger/much larger than actual download time
• Files are very small (<300 kb)
• Overall speed increases as we increase channels

"Downvotes" are:

• Overall speed does not change/drops when we increase channels (assuming file sizes stay mostly equal)
• Files are becoming too large and channel amount effect becomes not noticable
• TTFB increases or server start to respond slowly (increases "Downvote")
• Disk load on client increases and become slow to respond

The more "upvotes" we got, the more channels we create. "Downvotes" decrease amount of channels we need to keep open. We should start with 4 channels open.

And after second throught, sv_downloadurl_connections seems to be a bit useless for me now, since http web server gonna deliver only static files, and it should not experience any load problems with amount of channels. And, in fact, fastdl web server should be already capable of handling large amount of traffic.

[DBotThePony]

Almost 5 years after, I consider the idea of separate convars to control "download streams/channel" generally bad and HTTP/2 capable download client be the only solution to problem of RTT/TTFB.

Why? Because manual pipelining create issues, ranging from flow control problems and channel overload down to no sane way to control server load from HTTP's webserver perspective.

What should be done is implementing fully capable HTTP/2 downloader, and leaving HTTP/1.1 downloader logic as-is.

HTTP/2 has advantages with no downsides of manual HTTP/1.1 pipelining:

1. Game server require no additional configuration (only possible change required is switching sv_downloadurl to point at HTTP/2 capable server);
2. Data flow control is still in control by remote server (due to single connection), which allows TCP Congestion Control to do its job at making sure download channel is utilized optimally;
3. HTTP servers are themselves in charge of how many download streams they allow, which provides us with advantage described in 4;
4. We can send up to allowed number of GET requests to remote server, allowing us to probe/download bzipped resources first, and with zero effective latency get responses whenever they exist or not.
5. Because we get multiple download streams, we eliminate huge RTT impact caused by downloading a lot of small files.