search

intake / intake-xarray

Intake plugin for xarray
https://intake-xarray.readthedocs.io/
BSD 2-Clause "Simplified" License
74 stars 36 forks source link

Migration from deprecated dask.bytes to fsspec.core #105

Closed alyberty closed 3 years ago

alyberty commented 3 years ago

dask.bytes.open_files is marked deprecated (https://docs.dask.org/en/latest/_modules/dask/bytes.html). The replacement is in fsspec.core.open_files. By importing fsspec.core, it should silence/fix the deprecation notice.

martindurant commented 3 years ago

Sorry, I don't immediately see what's different in upstream, why it would fail. Do you have time to investigate different versions of xarray?

alyberty commented 3 years ago

The tests fails in test_intake_server_tiff and test_intake_server_netcdf, which should be totally unrelated. Is there an easy way to test current master against the same tests? My guess is, that the problem is already in master and not a result of my proposed change.

martindurant commented 3 years ago

My guess is, that the problem is already in master and not a result of my proposed change.

That's very likely true; I just thought perhaps they happen to reproduce for you, since you have a development environment set up already.

martindurant commented 3 years ago

@d70-t @scottyhq , perhaps you know what's going on here?

d70-t commented 3 years ago

These are just some ideas:

There has been a recent change which teaches xarray to open datasets with consolidated metadata by default. I think, this is likely the reason for the failing tests. That way, xarray will likely try to access .zmetadata from the store. The test seems to crash because that file is being tried (on some server side?). I am also wondering if this plays a role. Because ds.to_zarr will now try to consolidate_metadata which in turn tries to re-open the dataset in consolidated mode, which may fail in serialize_zarr_ds if the store doesn't keep .zmetadata.