BobbyWibowo / lolisafe

Blazing fast file uploader and awesome bunker written in node! 🚀
MIT License
317 stars 56 forks source link

Explore more about parallel chunked uploads #346

Closed BobbyWibowo closed 2 years ago

BobbyWibowo commented 3 years ago

Back in June, I rewrote the logic for chunked uploads so that instead of rebuilding their chunks later, all chunked uploads would instead be streamed to a single temporary file as they were being uploaded, one by one (https://github.com/BobbyWibowo/lolisafe/commit/b4c8b1d90e38bc517234a1aa58e7805b03e962ad). That had another advantage of being able to gradually hash the chunks as they were being uploaded, skipping the need to re-read the finished file merely for hashing (only then was it befitting of the whole "blazing fast" train of thought). But that was only feasible since chunked uploads were forcefully made to be uploaded in series. You could have multiple chunked uploads active all at once, but only when all of them were of different files.

BobbyWibowo commented 1 year ago

Dropped. Not possible to upload multiple chunks of the same file in parallel, if using a single destination file.