search

ESPRI-Mod / synda

ESGF Downloader (this is a deprecated repository, the tool has now moved to https://github.com/ESGF/esgf-download)
https://espri-mod.github.io/synda/
21 stars 11 forks source link

limits of sqlite? #69

Closed cehbrecht closed 3 years ago

cehbrecht commented 7 years ago

I just started to use synda ... have no experience here ... just guessing :)

synda uses the sqlite database. I have noticed that it protocols each downloaded file in the sqlite database. I would guess that if you keep only one instance of synda to replicate cmip6 with millions of files it might slow down ... just a guess, not sure about it.

You don't use SQLAlchemy? If you would do so, one would have the option to use a different database like postgres.

Or maybe using some kind of key-value store?

SebastienDenvil commented 7 years ago

We might decide to optimize this but haven't had any performance issue up to now. We replicated CMIP5 and CORDEX.

sqlite limits seem hard to reach. https://sqlite.org/limits.html

pjournou-ipsl commented 3 years ago

We have this implementation in our TODO list (probably available in 2022)