TUW-GEO / geopathfinder

Querying and searching data on the file system
MIT License
0 stars 2 forks source link

Subpath Crawler #5

Open wpreimes opened 5 years ago

wpreimes commented 5 years ago

In order to automate data updates, some of our data packages (gldas, ecmwf_models, ...) should contain functions to go through existing data structures and determine what data (start date, end date etc.) is already stored and what data is missing. E.g. in the gldas package there are functions for that https://github.com/TUW-GEO/gldas/blob/75ca48f620c1b64d7c6246f081aaa6924834b7ff/gldas/download.py#L43 and https://github.com/TUW-GEO/gldas/blob/75ca48f620c1b64d7c6246f081aaa6924834b7ff/gldas/download.py#L119

Without looking too much into this package now, is this something that could fit here? It would be nice if I don't have to add functions as the ones above to all our packages because that would mean a lot of duplicate code.

sebhahn commented 5 years ago

I think this is the right place