search

jimmywarting / StreamSaver.js

StreamSaver writes stream to the filesystem directly asynchronous
https://jimmywarting.github.io/StreamSaver.js/example.html
MIT License
3.95k stars 413 forks source link

Use Case: fetching many files from s3 and generating zip #291

Open gabriels1234 opened 1 year ago

gabriels1234 commented 1 year ago

I'm building a solution that requires broad compatibility. Got a few questions. @jimmywarting I was able to test downloading many files and getting them as a zip. This saves a lot of processing power/server costs, since there's nothing to do with the files, just give to the user as a zip. I was also able to plug this into a react app.

I have these questions, base on comments seen from @jimmywarting : 1) what improvements are there in native-file-system-adapter that pertain to fetching remote files. 2) Is it possible to use code from that repo to make StreamSaver not have the need for the mitm html without losing in features/compatibility? (namely, a cleaner solution) 3) In what cases the files fetched will be sitting on RAM? how to detect/know?

I know that at some point this will not the way of getting to this solution, but I must aim for maximum compatibility

Thanks!

gabriels1234 commented 1 year ago

UPDATE: I was able to make the download and zip of many files (say: 3000 100kb files) in parallel not using await for each fetch. initially, it caused the ERR::INSUFFICIENT..., and then I made it so each promise returned by the fetch was put in an array of promises. So, for every 1 MB of data downloaded, I awaited (await Promise.all(allPromises)). It made the trick.