search

luckman212 / jmap-backup

Back up a Fastmail JMAP mailbox in .eml format
https://news.ycombinator.com/item?id=41491351
16 stars 1 forks source link

TODO: multithreaded fetching #5

Open luckman212 opened 2 months ago

luckman212 commented 2 months ago

Add configurable # of download threads to increase speed. Not sure what Fastmail's ratelimits are, so 1-4 would probably be a good starting point.

alexpovel commented 1 month ago

Could perhaps use async! TaskGroup will allow the creation of a bunch of tasks to be run concurrently (similar to Go's WaitGroup or Rust's std::thread::scope if you're familiar) leveraging a context manager, awaiting them all. A task could be "backup an email". An asyncio.Semaphore can then help remain below rate limits, as a first approximation... it would at least help only run n (1-4) tasks concurrently.

All of that is single-threaded, but so are threads in Python (currently :-) ). In any case, it's more than fast enough, and you can easily reach hundreds of concurrent requests per second (but that's what the semaphore is for, limiting it).

luckman212 commented 1 month ago

@alexpovel Thank you for the suggestions. I plan to put some work into this next weekend. Hopefully will have a new version available for testing within a week or so.