NCAR / esm-collection-spec

Earth System Model Collection specification

Apache License 2.0

13 stars 7 forks source link

Amend JSON schema #30

Closed andersy005 closed 2 years ago

andersy005 commented 4 years ago

27 was merged by accident, and this PR amends it.

rabernat commented 4 years ago

I'd love to help revive this. Could someone do me the favor of providing a quick summary of where we stand with the esm-collection STAC extension?

andersy005 commented 4 years ago

I'd love to help revive this. Could someone do me the favor of providing a quick summary of where we stand with the esm-collection STAC extension?

Thank you for your patience, @rabernat! Regarding the summary, it's my understanding that 80%-90% of the work needed to bring this effort to completion is done, and the critical pieces missing here are:

formal validation of the schema against STAC
finalizing the specification document: https://github.com/andersy005/esm-collection-spec/blob/update-stac-spec/esm-collection/README.md

andersy005 commented 4 years ago

P.S. I'm going to work on pushing this forward this week

rabernat commented 4 years ago

I'd suggest we discuss a bit before moving forward. Things are evolving rapidly on CMIP6 cloud stuff. See e.g. discussion at https://github.com/pangeo-forge/cmip6-pipeline/issues/7.

I'm no longer sure we really need esm-collection. I think we can just index our cloud data into a hierarchical STAC catalog. Example here: https://nbviewer.jupyter.org/github/pangeo-forge/cmip6-pipeline/blob/master/catalog_stuff/crawl-serial.ipynb

Once we have such a collection, we could provide search capabilities via the STAC search API.

The CMIP6 cloud data group has been meeting biweekly on Fridays at 10am ET. We are meeting this Friday. @andersy005 I just invited you if you want to join. Your input would be very welcome.

andersy005 commented 4 years ago

I'm no longer sure we really need esm-collection. I think we can just index our cloud data into a hierarchical STAC catalog. Example here: nbviewer.jupyter.org/github/pangeo-forge/cmip6-pipeline/blob/master/catalog_stuff/crawl-serial.ipynb

This looks very promising! I was able to run the notebook locally.

We are meeting this Friday. @andersy005 I just invited you if you want to join. Your input would be very welcome.

Thank you for the invite, @rabernat! I'll be there. In the meantime, I am curious..

What will be the new role of intake-esm and/or intake-stac under this new approach?
Will this new approach be flexible enough to support assets in non-cloud native formats such as netCDF/HDF?

Once we have such a collection, we could provide search capabilities via the STAC search API.

👍 that'd be a very nice feature