search

NCAR / esm-collection-spec

Earth System Model Collection specification
Apache License 2.0
13 stars 7 forks source link

Add spec validator script #2

Closed andersy005 closed 4 years ago

andersy005 commented 4 years ago

CSV: use https://github.com/di/vladiate

rabernat commented 4 years ago

Could probably adapt many ideas from stac validator for the json part: https://github.com/sparkgeo/stac-validator

andersy005 commented 4 years ago

I am planning on making progress on this and #5 today, and tomorrow.

rabernat commented 4 years ago

FYI, I am hoping we can have something up and running by Wednesday so we can start writing the hackathon user guide.

I'm standing by to help however I can.

rabernat commented 4 years ago

We should also engage with @martindurant to figure out the best way to bring the qgrid-style catalog browsing into intake (as he demo'd in https://github.com/intake/intake/issues/417).

andersy005 commented 4 years ago

@naomi-henderson

Is https://storage.googleapis.com/cmip6/cmip6-zarr-consolidated-stores.csv still the most recent cmip6 google cloud csv catalog?

@rabernat, where should we store the actual CMIP6 Google Cloud collection (json file)?

For CMIP6 on Glade, we've been granted permission to glade/collections/cmip/catalog and I am planning on setting up a cron-job to update the (csv) catalog over night.

Cc @matt-long

naomi-henderson commented 4 years ago

Yes, @andersy005 , that is the csv catalog file that reflects our current collection. The column with the path to the zarr store is still called zstore rather than path - do we still want to change it?

andersy005 commented 4 years ago

do we still want to change it?

With the new esm-collection-spec, we don't need to change it. We will use it as is by specifying column_name to be zstore in the collection-spec file.

andersy005 commented 4 years ago

Implemented in #9. Needs improvements though ( I will look into this in a separate PR)