search

ESPRI-Mod / synda

ESGF Downloader (this is a deprecated repository, the tool has now moved to https://github.com/ESGF/esgf-download)
https://espri-mod.github.io/synda/
21 stars 11 forks source link

interleaving of jobs #73

Closed painter1 closed 3 years ago

painter1 commented 7 years ago

It is normal for Synda to be running a few jobs (each defined by "synda install -s ...") at once. When running two jobs, I have seen (with "synda watch") that sometimes only one job is active, i.e. all files being downloaded, belong to that one job.

Sometimes one server is much slower than another one. So good performance requires that each job has a few, e.g. 4, files being actively downloaded at any time.

I suggest that the parameter max_parallel_download apply separately to each job.

pjournou-ipsl commented 3 years ago

In the future 3.4 release, a new feature for asynchronous downloads will be implemented. It is built on the asyncio library. We measured higher performances that we hope near to be the best that can be reached.