ESPRI-Mod / synda

ESGF Downloader (this is a deprecated repository, the tool has now moved to https://github.com/ESGF/esgf-download)
https://espri-mod.github.io/synda/
21 stars 11 forks source link

Per-datanode maximum on parallel downloads #103

Closed painter1 closed 5 years ago

painter1 commented 5 years ago

In large-scale replication work, there often is a problem where all permitted downloads are taken up by a slow server with big files. But to get a good transfer rate, you must keep all data nodes active at all times. The simple way to do this is to set a maximum number of parallel downloads separately for each data node. This does not preclude setting an overall maximum as before.