Kataiser
commented
5 years ago gzip sounds great, but at least in my case, it only decreases the file size by about 25% (9.6 MB to 7.2 MB), whereas LZMA is by about 80% (to 1.7 MB). Granted, this is in 7zip with the maximum Deflate dictionary size of only 32 KB. Although implicit clientside compression is super cool, maybe there's a library somewhere that can do a better algorithm, or at least better settings.
Uploads to Splits I/O over unreliable internet can take multiple attempts, see: https://twitter.com/GmMecharon1/status/1093361399663865856
Supporting compression formats is one way to go about this, and probably the simplest to implement for now. The user drag-and-dropping ZIP files is one thing, but ideally this happens without changing user behavior -- submitting an upload first makes JS compress the splits in whatever format (gzip?) and upload to S3, where it's decompressed (does S3 have implicit support for this?).
In the future this could be differently handled by parsing runs locally and uploading them piecemeal, i.e. one segment / history / etc. at a time. This is likely to be eventually supported on the backend anyway because of our desire to eventually allow runs to be uploaded in realtime as they happen, piece by piece. When that happens, maybe it's time to think about frontend parsing. But that's totally not a requirement for solving this issue, just something to keep in mind for the future.