Speed up atlas downloads

brainglobe / brainglobe-atlasapi

A lightweight python module to interact with atlases for systems neuroscience

https://brainglobe.info/documentation/brainglobe-atlasapi/index.html

BSD 3-Clause "New" or "Revised" License

127 stars 34 forks source link

Speed up atlas downloads #407

Open adamltyson opened 2 months ago

adamltyson commented 2 months ago

Downloading the atlases can take a long time. In some areas it can seemingly take hours. We should investigate alternatives to improve this, e.g.:

Move to other providers (e.g. Zenodo https://github.com/brainglobe/brainglobe-atlasapi/issues/284#issuecomment-2335129409)
Region-specific mirrors (cc @dbirman)

adamltyson commented 1 month ago

Just to note, there are lots of places we could store the data, but with various caveats.

Some data sharing repositories - most are not suitable for this type of data, e.g. because the data needs to meet certain standards, and pass a curation process.
AWS/cloudflare etc - potentially unpredictable costs
EBRAINS - there's an object store (in addition to the curated knowledge base)

NicoKiaru commented 1 month ago

Hi! Just to report the speed in Switzerland / Lausanne: it can go as slow at 1 MB/s

alessandrofelder commented 1 month ago

Thanks for reporting @NicoKiaru - hopefully we can improve on this in the near future.

NicoKiaru commented 1 month ago

Sure, I don't want to make you feel that this post is to put pressure - it's just to be informative and report typical DL speed. I mean, waiting 15 minutes once is not a big problem.