Integration with Pangeo data

[meggart]

Now that we are able to read xarray-flavored zarr into our cube format, it should be possible to directly process the Pangeo-hosted CMIP6 data. Given that the PR at AWSCore gets merged, there should be no obstacles. Here is a short gist https://gist.github.com/meggart/2d544be2c1368f8774d0a21ea4633985 showing some (meaningless) processing on the Pangeo data done from my laptop. Of course this assumes the respective branches of AWSCore and Zarr are installed.

@rabernat from your comments in other threads it looks like you are interested in making the Pangeo data accessible to Julia users, maybe you could comment if this would be of any use for you?

[rabernat]

Julia support is very important to Pangeo. The principles that we are trying to advocate:

• analysis ready data
• data proximate Computing
• interactive analysis at scale
• open source and reproducibility

are in no way specific to python.We see our current stack (xarray, dask, zarr) as just one implementation.

Let us know how we can help!

[meggart]

Thanks a lot for your support in getting the AWSCore PR merged. I am slowly adapting this into downstream packages now, Zarr.jl is already updated and will be registered soon. I have added a small example to the docs how to access the CMIP6 data as well https://meggart.github.io/Zarr.jl/latest/s3examples/#Accessing-CMIP6-data-on-GCS-1

When the new version of Zarr is tagged it will also be possible to interact with the data in a more convenient way as show in the gist through this package.

[rabernat]

Fabian, this is great! Do you mind if I link to your example from the GCP official docs for CMIP6?

[meggart]

I don't mind, please go forward.