ESPRI-Mod / synda

ESGF Downloader (this is a deprecated repository, the tool has now moved to https://github.com/ESGF/esgf-download)
https://espri-mod.github.io/synda/
21 stars 11 forks source link

Special values of max parallel downloads #163

Open painter1 opened 3 years ago

painter1 commented 3 years ago

Sometimes, notably for CCCma, a data node enforces a maximum number of simultaneous connections, which differs from what we would want for most data nodes. To transfer data from such a node, we need a special node-dependent value of max_parallel_download. That is what this patch enables. To use it, set max_parallel_download_special_datanodes in the download section of the configuration file. The value is a comma-separated list of special datanodes and values (separated with a colon) such as: max_parallel_download_special_datanodes = crd-esgf-drc:3, tropmet:1 Note that the full datanode url or name is not required; you only need a substring.