Is your feature request related to a problem? Please describe.
The current download protocol uses two TCP sockets in two threads (one thread per socket) to send mod files in chunks of some size (like 100 MB or so) per thread. When $M$ people join, two threads exist per player, which leads to $M$ additional threads spawned. In total, there are $2M$ threads, each allocating a chunk of size $C$, resulting in memory usage of $2MC$ bytes. For example 10 players joining, at a 100 MB chunk size, results in $10*2\100=2000$ bytes, or 2 GB, of memory usage. Correct my math if I'm wrong, or if I misunderstand the code.
Not only is downloading across two sockets not necessarily more efficient than one socket, but also the code is not thread safe (at least on the launcher side).
This whole thing is being replaced with the new protocol anyways, where we just use sendfile() to send the file. We can adopt this approach (see the new-protocol branch), or send in chunks if we want to keep track of progress.
Describe the solution you'd like
[ ] Use a single thread and the existing TCP connection to download the files over. Use either an interface like sendfile() or manual chunking (ew ew ew ew) to send the file efficiently, and test it on a 1 Gbps link. A single TCP connection should easily saturate that.
[ ] Implement a timeout, I'm ok if this is per chunk (ew), but better something builtin (yay) like via boost::asio. Obviously this timeout CANNOT be some magic number, it needs to allow for shitty slow antarctic internet of like 200 KBps, as long as bytes are flowing, keep it going. We can share the memory for the same mods to make it difficult to DoS the server by opening download connections, or something, probably?
[ ] Either don't use heap allocations (by using kernel buffers, like sendfile() does), or make the size of the total allocations across the entire server have an adjustable maximum.
Describe alternatives you've considered
HTTP based download, but that's not trivial to do over the same socket. All other alternatives were probably considered and likely are gonna be implemented for the new protocol.
Additional context
The current implementation is probably the oldest part of the codebase. Here be dragons and so on.
Is your feature request related to a problem? Please describe. The current download protocol uses two TCP sockets in two threads (one thread per socket) to send mod files in chunks of some size (like 100 MB or so) per thread. When $M$ people join, two threads exist per player, which leads to $M$ additional threads spawned. In total, there are $2M$ threads, each allocating a chunk of size $C$, resulting in memory usage of $2MC$ bytes. For example 10 players joining, at a 100 MB chunk size, results in $10*2\100=2000$ bytes, or 2 GB, of memory usage. Correct my math if I'm wrong, or if I misunderstand the code.
Not only is downloading across two sockets not necessarily more efficient than one socket, but also the code is not thread safe (at least on the launcher side).
This whole thing is being replaced with the new protocol anyways, where we just use sendfile() to send the file. We can adopt this approach (see the new-protocol branch), or send in chunks if we want to keep track of progress.
Describe the solution you'd like
boost::asio. Obviously this timeout CANNOT be some magic number, it needs to allow for shitty slow antarctic internet of like 200 KBps, as long as bytes are flowing, keep it going. We can share the memory for the same mods to make it difficult to DoS the server by opening download connections, or something, probably?Describe alternatives you've considered HTTP based download, but that's not trivial to do over the same socket. All other alternatives were probably considered and likely are gonna be implemented for the new protocol.
Additional context The current implementation is probably the oldest part of the codebase. Here be dragons and so on.