Notebooks cannot access the data

[brian-rose]

Currently the notebooks are failing at the intake stage with

ValueError: Bad Request: https://storage.googleapis.com/download/storage/v1/b/pangeo-cmems-duacs/o/.zmetadata?alt=media
User project specified in the request is invalid.

e.g. here: http://cookbooks.projectpythia.org/physical-oceanography-cookbook/01_sea-surface-height.html#initialize-dataset

[brian-rose]

It appears that the datasets are in "requester pays" storage, so some credentials are needed to handle billing for data egress: https://catalog.pangeo.io/browse/master/ocean/sea_surface_height/

[brian-rose]

Are any of the datasets available here? https://pangeo-forge.org/catalog without egress fees?

[mgrover1]

As was mentioned during the meeting, the next steps would be to:

• [ ] Replicate the pangeo catalog in this repo (link to catalog)
• [ ] Replace the data links with data from the open storage network (OSN) (list of datasets on pangeo-forge)
• [ ] Upload the updated catalog to GitHub
• [ ] Use the new catalog in the notebooks

[mgrover1]

One other option for a data source is the ERA5 data on Amazon which is freely available https://registry.opendata.aws/ecmwf-era5/, with the example notebook here

[r-ford]

Copernicus has some tutorials like this one that go through how to access the data. It requires credentials, so I'm not sure if this is an option for us.

[mgrover1]

There is a way to add credentials, it requires some Github secrets configuration