[x] Test and copy LEAP stores (one time action, info -> 'leap-pangeo.cmip6_pgf_ingestion.leap_legacy'
[x] Test PANGEO legacy stores (one time action, info -> 'leap-pangeo.cmip6_pgf_ingession.pangeo_legacy'
[x] Add all legacy entries to beta test table "leap-pangeo.testcmip6.cmip6_consolidated_manual_testing"
[x] Set up revised json catalog files
Beta version of catalog available here:
import intake
url = "https://storage.googleapis.com/cmip6/cmip6-pgf-ingestion-test/catalog/catalog.json" # Only stores that pass current tests
# url = "https://storage.googleapis.com/cmip6/cmip6-pgf-ingestion-test/catalog/catalog_noqc.json" # Only stores that fail current tests
# url = "https://storage.googleapis.com/cmip6/cmip6-pgf-ingestion-test/catalog/catalog_retracted.json" # Only stores that have been retracted by ESGF
col = intake.open_esm_datastore(url)
Once we moved the ingestion to the public GCS bucket the instructions in the README should be updated and this issue closed.
Some steps to track the evolution of this milestone. We are getting VERY CLOSE.
[x] Implement mover stage (#67)
[x] Test access to public bucket from mover stage (#87)
[x] Refactor all database content to a single bigquery table:
[x] Test and copy LEAP stores (one time action, info ->
'leap-pangeo.cmip6_pgf_ingestion.leap_legacy'
[x] Test PANGEO legacy stores (one time action, info ->
'leap-pangeo.cmip6_pgf_ingession.pangeo_legacy'
[x] Add all legacy entries to beta test table
"leap-pangeo.testcmip6.cmip6_consolidated_manual_testing"
[x] Set up revised json catalog files Beta version of catalog available here:
[ ] Get some folks to beta test.
[ ] Rebase on
leap-data-management-utils
release (#85)[ ] Implement consolidated coordinates (#88)
[ ] Optional/Later: Consolidate coordinates of LEAP legacy stores?
[ ] Setup final table and repeat ingestion from legacy bq tables.
'leap-pangeo.cmip6_pgf_ingestion.leap_legacy'
'leap-pangeo.cmip6_pgf_ingession.pangeo_legacy'
[ ] Delete and clean temp table/directories and populate final table from scratch.
cmip6/cmip6-pgf-ingestion-test/catalogs/
cmip6/cmip6-pgf-ingestion-test/
cmip6/CMIP6_LEAP_legacy_test
[ ] Remove retraction logic from https://github.com/pangeo-data/pangeo-cmip6-cloud/blob/master/create_filtered_catalog.py and document new location there.
[ ] Adopt new catalog in xMIP
[ ] Write a Blogpost about this
[ ] Dance in the streets 🕺
[ ] (Optional/Later) Implement regular retesting of all stores