data discovery : file system globbing may be too expensive

rigoudyg / climaf

CliMAF - a Climate Model Analysis Framework - doc at : http://climaf.readthedocs.org/

Other

15 stars 7 forks source link

data discovery : file system globbing may be too expensive #102

Open senesis opened 5 years ago

senesis commented 5 years ago

Should develop a way to read a file system 'audit file', in order to avoid globbing too large directories, as e.g. /bdd on Ciclad

jservonnat commented 5 years ago

Potentially project-dependant?

senesis commented 5 years ago

Rather directory dependant because system managers, who should be in charge of cron-ing the audit file generation, are not 'project-aware', but 'directory-aware'. CliMAF would just, when using an 'url', search for an audit file for each prefix in the URL (up to the point where it finds a wildcard). So, we just need a convention on :

location of an audit file for a given directory
content of the audit file (albeit there is some natural choice, here)