muxinc / upchunk

Uploads Chunks! Takes big files, splits them up, then uploads each one with care (and PUT requests).
MIT License
336 stars 46 forks source link

Add dynamic chunk sizing #90

Closed jaredsmith closed 2 years ago

jaredsmith commented 2 years ago

This adds dynamic chunk sizing to upchunk.

There are three new options:

The algorithm is somewhat primitive, but seems to work quite well in practice. It starts off using the chunk size defined by chunkSize, and if that chunk takes less than 10 seconds, it doubles the size. If a chunk takes more than 30 seconds, it halves the chunk size. (Of course, looking out for minimums and maximums.)

jaredsmith commented 2 years ago

Converting back to a draft, as I need to fix up the tests now.