@agstephens attended the ESMValTool Workshop at the Met Office in October 2022. He presented on the current state of the STAC-for-ESGF developments at CEDA.
This issue contains a list of points/questions that were raised during the talk - they should be ticked off when a separate GitHub issue has been created for each.
Issues raised
[ ] Make sure our collections have titles - this should be on the server side.
[ ] Put in a PR for our pystac_client changes.
[ ] Re-test indexing this dataset: CMIP6.HighResMIP.MOHC.HadGEM3-GC31-HH.highres-future.r1i1p1f1.day.tas.gn.v20191105 - should have 36 files (assets) and no duplicates!
[ ] Investigate problems with bbox search
[ ] Create a version of the notebook(s) that work in binder - and send out for review.
[ ] Solr records in ESGF often have incorrect content. We, therefore, recommend that all datasets are re-scanned and republished from the file-system level rather than migrating content from Solr to STAC.
[ ] How does the latest boolean option work in the search system? Are records changed in Solr? Or is it all managed by the server-side layer? Need to replicate the behaviour in STAC.
[ ] SMHI has stated that it might want to publish "local" datasets. These would not necessarily be of interest to ESGF globally. How can we manage this kind of requirement? Should there be a set of federated nodes or just one search node?
[ ] Ask Bouwe for 3 different ESMValTool recipes that we can develop search code for in our client. We can use these as use cases, unit tests, and use them to benchmark response times when comparing Solr and STAC.
[ ] LdM mentioned the need for complex queries such as "Get me all the model/expt/ensembles that have ALL 3 OF THESE VARIABLES present" - is there a way to construct such a query?
[ ] Make sure that version works as a queryable facet - as it should.
[ ] Create a new and simple notebook to show the functionality.
[ ] Replicas - how are they, and how should they be managed? Does our index treat them properly?
[ ] Make sure pagination works - with items and assets.
Notes
In our current implementation, the asset has replica, dataset_version and latest properties - because we have harvested it straight from Solr. See:
Overview
@agstephens attended the ESMValTool Workshop at the Met Office in October 2022. He presented on the current state of the STAC-for-ESGF developments at CEDA.
This issue contains a list of points/questions that were raised during the talk - they should be ticked off when a separate GitHub issue has been created for each.
Issues raised
pystac_clientchanges.CMIP6.HighResMIP.MOHC.HadGEM3-GC31-HH.highres-future.r1i1p1f1.day.tas.gn.v20191105- should have 36 files (assets) and no duplicates!bboxsearchlatestboolean option work in the search system? Are records changed in Solr? Or is it all managed by the server-side layer? Need to replicate the behaviour in STAC.versionworks as a queryable facet - as it should.itemsandassets.Notes
In our current implementation, the
assethasreplica,dataset_versionandlatestproperties - because we have harvested it straight from Solr. See:https://api.stac.ceda.ac.uk/asset/search?limit=1
The
itemonly hasdataset_version. It is missingreplicaandlatest- which arguably belong more in theitemthan in theasset.