bopen / xarray-sentinel

Xarray backend to Copernicus Sentinel-1 satellite data products
Apache License 2.0
219 stars 22 forks source link

Chunking works only with dask installed #50

Open corrado9999 opened 3 years ago

corrado9999 commented 3 years ago

I am not sure if this is a bug and, if so, of what.

>>> path = "tests/data/S1B_IW_SLC__1SDV_20210401T052622_20210401T052650_026269_032297_EFA4.SAFE/manifest.safe"
>>> group = 'IW1/R168-N471-E0118'
>>> xs.sentinel1.open_dataset(path, group=group)
<xarray.Dataset>
Dimensions:           (azimuth_time: 1501, slant_range_time: 21632)
Coordinates:
    line              (azimuth_time) float64 0.5 1.5 2.5 ... 1.5e+03 1.5e+03
    pixel             (slant_range_time) float64 0.5 1.5 ... 2.163e+04 2.163e+04
  * azimuth_time      (azimuth_time) datetime64[ns] 2021-04-01T05:26:24.20999...
  * slant_range_time  (slant_range_time) float64 0.005343 0.005343 ... 0.005679
Data variables:
    VH                (azimuth_time, slant_range_time) complex128 ...
    VV                (azimuth_time, slant_range_time) complex128 ...
Attributes: (12/14)
    constellation:              sentinel-1
    platform:                   sentinel-1b
    instrument:                 ['c-sar']
    sat:orbit_state:            descending
    sat:absolute_orbit:         26269
    sat:relative_orbit:         168
    ...                         ...
    sar:instrument_mode:        IW
    sar:polarizations:          ['VV', 'VH']
    sar:product_type:           SLC
    xs:instrument_mode_swaths:  ['IW1', 'IW2', 'IW3']
    Conventions:                CF-1.7
    history:                    created by xarray_sentinel-999

>>> xs.sentinel1.open_dataset(path, group=group, chunks=1000)
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-5-79175c680e53> in <module>
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-7-90f5963f6ea0> in <module>
----> 1 xs.sentinel1.open_dataset(path, group=group, chunks=1000)

~\xarray-sentinel\xarray_sentinel\sentinel1.py in open_dataset(product_urlpath, drop_variables, group, chunks, fs)
    384         else:
    385             annotation_path = list(groups[group]["annotation_path"].values())[0]
--> 386             ds = open_burst_dataset(
    387                 manifest_path,
    388                 measurement_paths=groups[group]["measurement_path"],

~\xarray-sentinel\xarray_sentinel\sentinel1.py in open_burst_dataset(manifest_path, burst_position, measurement_paths, annotation_path, chunks)
    297     data_vars = {}
    298     for pol, data_path in measurement_paths.items():
--> 299         arr = rioxarray.open_rasterio(data_path, chunks=chunks)
    300
    301         arr = arr.squeeze("band").drop_vars(["band", "spatial_ref"])

~\miniconda3\envs\xr-sentinel\lib\site-packages\rioxarray\_io.py in open_rasterio(filename, parse_coordinates, chunks, cache, lock, masked, mask_and_scale, variable, group, default_name, **open_kwargs)
    863
    864     if chunks is not None:
--> 865         result = _prepare_dask(result, riods, filename, chunks)
    866
    867     # Make the file closeable

~\miniconda3\envs\xr-sentinel\lib\site-packages\rioxarray\_io.py in _prepare_dask(result, riods, filename, chunks)
    573     """
    574     # pylint: disable=import-outside-toplevel
--> 575     from dask.base import tokenize
    576
    577     # augment the token with the file modification time

ModuleNotFoundError: No module named 'dask'
aurghs commented 3 years ago

We could: 1) Add can add a check to skip tests with dask if dask is not installed. 2) Improve the message error message. That should be done in rioxarray

alexamici commented 3 years ago

This is indeed the default behaviour of xarray. But I agree that the error message is awful.

@aurghs I propose to do in the reverse order:

  1. open a PR to rioxarray where it raises an exception with a better error message
  2. (if 1. fails) add the same check in xarray-sentinel
aurghs commented 3 years ago

This is indeed the default behaviour of xarray. But I agree that the error message is awful.

1 and 2 are not an aut-aut. I would:

alexamici commented 3 years ago
  1. we do install dask for testing already, no need to test without dask, we don't have any dask related logic to test. (I propose no action)
  2. dask is not a needed dependency so it must not be listed in the install_requires (no action)
  3. we agree, @aurghs would you please take care of it?
aurghs commented 3 years ago
  1. we could add an xfails on the tests with dask
  2. I agree
  3. I think @corrado9999 was preparing the PR.