I’m pretty new to boto3 and mostly following the template in the Jupyter notebook where images are downloaded one-by-one. This mostly seems ok download speed wise if I parallelize with multiple workers, but it seems to have a nasty habit of hanging at times and needing a reset – I’m not sure if I’m being throttled trying to access files systematically like this. Let me know if I am doing something terrible that I should not be doing, and if you’d have any better guidance.
We recommend doing this:
https://github.com/jump-cellpainting/2023_Chandrasekaran_submitted#step-1-download-cell-images