search

pangeo-data / pangeo-datastore

Pangeo Cloud Datastore
https://catalog.pangeo.io
48 stars 16 forks source link

Use consolidated metadata for all Zarr datasets #119

Closed charlesbluca closed 3 years ago

charlesbluca commented 3 years ago

This PR sets consolidated: True for Zarr datasets in our Intake catalogs, so that they can potentially be accessed by STAC Browser down the line.

Before this is merged, I would like to also consolidate the metadata for hydro.nldas_hru_forcing and hydro.usgs_streamflow_observations, which reside in S3 storage.

martindurant commented 3 years ago

Was there a reason that this PR stalled?

charlesbluca commented 3 years ago

If I recall, this was because I never actually got the write access to Pangeo's S3 buckets to consolidate the Zarr metadata there - I can double check that when I get the chance.

martindurant commented 3 years ago

Can you add the flag, though, to those that are already consolidated?

charlesbluca commented 3 years ago

Yes - just checked and currently all GCS Zarr datasets are now flagged as consolidated. This should be good to merge as is, and we can follow up on the S3 Zarr datasets later on.