Slow down RCSB download if request limit is reached

BorgwardtLab / proteinshake

Protein structure datasets for machine learning.

https://proteinshake.ai

BSD 3-Clause "New" or "Revised" License

99 stars 8 forks source link

Slow down RCSB download if request limit is reached #136

Open timkucera opened 1 year ago

timkucera commented 1 year ago

For now I just implemented a hard limit on the number of threads, but the download speed may vary between machines. We should check for the response code and delay the download if necessary