Add local database table for ESGF facets

ScottWales commented 5 years ago

Download the list of valid facets from ESGF to a local table.

This would allow facets to be validated for all projects, including CORDEX etc.

It would also allow use of SQL wildcards to select multiple projects, e.g. model='ACCESS%'

Requires:

[ ] New database table esgf_facet with columns key, value
[ ] Admin command to download facets from ESGF 'https://esgf.nci.org.au/esg-search/search/?offset=0&limit=0&facets=*' and add new entries to the database (on a cron job)
[ ] Add validation logic to the cli and api to expand wildcards and raise an error if a facet is not valid

paolap commented 5 years ago

In a way we have the first "dot" not as database table but as json files. What's still missing is the point 2 and 3

paolap commented 3 years ago

There is now a function in cordex.py get_esgf_facets(project) to retrieve the facets and values. Will need to review this when cordex release is done

paolap commented 2 years ago

We reverted to previous behaviour just getting the facets from json file. Will have to review this later on

coecms / clef