ESPRI-Mod / synda

ESGF Downloader (this is a deprecated repository, the tool has now moved to https://github.com/ESGF/esgf-download)
https://espri-mod.github.io/synda/
21 stars 11 forks source link

Apply the incremental mode when installing datasets #146

Closed painter1 closed 4 years ago

painter1 commented 4 years ago

If the user edits sdt.conf to set 'incremental_mode_for_datasets' to ……'true', then the incremental mode (synda install -i ...) will be applied to dataset searches as well as file searches. This has a huge performance advantage for large installation jobs, roughly a factor of 50 in my experience. That, in turn, reduces the number of database lock errors encountered by synda processes.

AtefBN commented 4 years ago

Thanks as always @painter1 !