Download files in parallel?

felixrieseberg / slack-archive

Generate static HTML archives of Slack workspaces

243 stars 33 forks source link

Open cheapsteak opened 2 years ago

cheapsteak commented 2 years ago

Sometimes there's a large file or a file on a slow server, clogs the downloads up quite a bit

Thoughts on going through all the channel data to create a queue, and going through the queue with some parallelism? (e.g. with p-limit)

felixrieseberg commented 2 years ago

I'm mostly worried about rate limits, which I'm already hitting quite a bit - but it's worth a try?