Break up intake-xarray into separate drivers?

[danielballan]

Intake-xarray lumps together several different I/O backends. Grouping drivers by what they return (xarray in this case) is different from the usual pattern. We propose to break intake-xarray into intake-netcdf, intake-tiff with rasterio support (and maybe also tifffile support?), intake-png, and intake-zarr. We can maintain intake-xarray going forward as a metapackage that depends on these as a convenience and for back-compat.

[martindurant]

You may well have a point, but the original rationale was: that all these call xarray loader function

[danielballan]

Yes, I think that is sensible early on, just as we currently package a bunch of unrelated drivers together in databroker._drivers with the intention of splitting them up once things stabilize. Maybe this is worth doing eventually, with some shared utility library containing the xarray loader.

[jbednar]

Seems fine to me...

[joshmoore]

Happy to write it up as a separate issue, but if this would help load zarrs which were not written with xarray, I'd also be in favor.

I'm currently running into a KeyError on _ARRAY_DIMENSIONS.

Stacktrace

``` Traceback (most recent call last): File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/zarr.py", line 163, in _get_zarr_dims_and_attrs dimensions = zarr_obj.attrs[dimension_key] File "/Users/jamoore/opt/zarr/zarr/attrs.py", line 64, in __getitem__ return self.asdict()[item] KeyError: '_ARRAY_DIMENSIONS' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "intake_test.py", line 3, in ds = cat.idr_ebi_6001240.to_dask() File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/intake_xarray/base.py", line 69, in to_dask return self.read_chunked() File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/intake_xarray/base.py", line 44, in read_chunked self._load_metadata() File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/intake/source/base.py", line 117, in _load_metadata self._schema = self._get_schema() File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/intake_xarray/base.py", line 18, in _get_schema self._open_dataset() File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/intake_xarray/xzarr.py", line 31, in _open_dataset self._ds = xr.open_zarr(self._mapper, **self.kwargs) File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/zarr.py", line 599, in open_zarr ds = maybe_decode_store(zarr_store) File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/zarr.py", line 582, in maybe_decode_store drop_variables=drop_variables, File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/conventions.py", line 570, in decode_cf vars, attrs = obj.load() File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/common.py", line 123, in load (_decode_variable_name(k), v) for k, v in self.get_variables().items() File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/zarr.py", line 290, in get_variables (k, self.open_store_variable(k, v)) for k, v in self.ds.arrays() File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/core/utils.py", line 402, in FrozenDict return Frozen(dict(*args, **kwargs)) File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/zarr.py", line 290, in (k, self.open_store_variable(k, v)) for k, v in self.ds.arrays() File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/zarr.py", line 274, in open_store_variable dimensions, attributes = _get_zarr_dims_and_attrs(zarr_array, _DIMENSION_KEY) File "/opt/anaconda/envs/py36/lib/python3.6/site-packages/xarray/backends/zarr.py", line 167, in _get_zarr_dims_and_attrs "required for xarray to determine variable dimensions." % (dimension_key) KeyError: 'Zarr object is missing the attribute `_ARRAY_DIMENSIONS`, which is required for xarray to determine variable dimensions.' ```

[martindurant]

load zarrs which were not written with xarray

LIke intake.source.zarr.ZarrArraySource ("ndzarr")?

[joshmoore]

Thanks, @martindurant. ZarrArraySource does work for my data.