Closed aradhakrishnanGFDL closed 1 year ago
@aradhakrishnanGFDL - I had a chance to look at this, and there was a bug in how we are passing storage_options
to non-zarr formats (ex. netcdf4)
With the fix over in #609 , you should be able to read the dataset using the following snippet
dset_dict = cat.to_dataset_dict(xarray_open_kwargs={'engine':'h5netcdf',
'chunks':{}},
storage_options={'anon':True},
)
We will cut a release in the next few days with the fix once it is merged!
Thank you for raising this issue! 🎉
I used the pangeo docker images pangeo/pangeo-notebook:2021.01.07 successfully to run an old notebook that uses intake-esm to access CMIP6 S3 netcdf data in AWS [open datasets, anon=true].
With the latest images, they don't work as expected and I tried to run stand alone tests with intake-esm. Hope the following helps in tracing the issue with the package or the usage
With intake-esm, version 2020.12.18 (And s3fs 2023.4.20), this notebook works as expected.
With 2023.4.20 and probably starting some version in 2022?, the notebook breaks here
dset_dict = cat.to_dataset_dict(cdf_kwargs={'chunks': {'time': 1}},storage_options={'anon':True})
This snippet from the notebook may help reproduce the error:
ERRORs