neon_index() should support filters on filename metadata, returning only files which include
[x] requested product(s)
[x] requested table(s)
[x] requested site(s)
[x] requested month[s] (probably in start_date / end_date format)
[x] requested file types
...
These should be inherited by neon_store() and neon_read().
The latter is particularly important, as metadata from the file name may not also be included in the table itself, notably, siteID. (e.g. In this case, users could at least apply neon_read() site-by-site and then add the site info).
Filtering prior to read will also be important when the data gets very large!
cboettig
commented
4 years ago
neon_index()should support filters on filename metadata, returning only files which includeThese should be inherited by
neon_store()andneon_read().The latter is particularly important, as metadata from the file name may not also be included in the table itself, notably, siteID. (e.g. In this case, users could at least apply
neon_read()site-by-site and then add the site info).Filtering prior to read will also be important when the data gets very large!
see: https://github.com/cboettig/neonstore/blob/master/notebook/read_with_file_metadata.R for current work-around using the low-level interface (
neon_index()).Thanks @rqthomas for this.