search

NCAS-CMS / cfstore

A CF compliant data store with interfaces to local and remote storage.
MIT License
2 stars 1 forks source link

Coordinate B metadata extractor and support for B metadata in database. #37

Open bnlawrence opened 2 years ago

bnlawrence commented 2 years ago

What will we do about coordinate metadata? Obviously values in aggregation files, but do we want ISO like bounding boxes (is there any use for mostly global gridded data? Is what we really need at this point a "resolution" item?

bnlawrence commented 2 years ago

Starting to think about this. Planning on a key-value table and a json blob table to be attached to all files. Standard NC variables go into the key-value table and we can do something more sophisticated with the other stuff using JSON.

(NB: Expecting the CFA aggregation to mean the number of files we need to hold in any user instance of cfstore will not scale to millions.)

bnlawrence commented 2 years ago

@davidhassell Do you think we even need the JSON blob? Anything using actual coordinates for queries probably ought to operate on the netcdf itself? (Or hide in another sort of database which could be opened in parallel with the text database?)

bnlawrence commented 2 years ago

So, yeah, we probably do need to have the JSON blob, otherwise we have to support all the various types of netcdf attributes with their own tables. There's a balance to be struck here ....

bnlawrence commented 2 years ago

the issue25 branch now has a reasonably complete implementation of a minimal b-metadata ingest. It'd be good to discuss how to minimally represent a CF_DOMAIN construct - bounding box, size, resolution etc. @davidhassell we need to talk about this ...

bnlawrence commented 2 years ago

Do we also want to consider an inner and outer bounding box for the projection of a rotated grid onto the sphere?

bnlawrence commented 1 year ago

We don't really need this for the MVP, it's really going to be useful when we have CANARI regional model data to handle.