makepath / xarray-spatial

Raster-based Spatial Analytics for Python
https://xarray-spatial.readthedocs.io/
MIT License
844 stars 85 forks source link

getting error when trying to run look <source elided> out = np.full(arr1.shape, np.nan, dtype=np.float32) rows, cols = arr1.shape #802

Open krishnaglodha opened 6 months ago

krishnaglodha commented 6 months ago

Describe the bug I'm trying to get statistical values of my xarray.DataArray in loop , but getting the error

TypingError                               Traceback (most recent call last)
Cell In[18], [line 2](vscode-notebook-cell:?execution_count=18&line=2)
      [1](vscode-notebook-cell:?execution_count=18&line=1) for val in index_xarray:
----> [2](vscode-notebook-cell:?execution_count=18&line=2)     print(val.max().compute().values.item())

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1169, in DataArray.compute(self, **kwargs)
   [1150](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1150) """Manually trigger loading of this array's data from disk or a
   [1151](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1151) remote source into memory and return a new array. The original is
   [1152](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1152) left unaltered.
   (...)
   [1166](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1166) dask.compute
   [1167](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1167) """
   [1168](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1168) new = self.copy(deep=False)
-> [1169](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1169) return new.load(**kwargs)

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1143, in DataArray.load(self, **kwargs)
   [1125](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1125) def load(self, **kwargs) -> Self:
   [1126](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1126)     """Manually trigger loading of this array's data from disk or a
   [1127](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1127)     remote source into memory and return this array.
   [1128](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1128) 
   (...)
   [1141](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1141)     dask.compute
   [1142](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1142)     """
-> [1143](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1143)     ds = self._to_temp_dataset().load(**kwargs)
   [1144](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1144)     new = self._from_temp_dataset(ds)
   [1145](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataarray.py:1145)     self._variable = new._variable

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:845, in Dataset.load(self, **kwargs)
    [842](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:842) chunkmanager = get_chunked_array_type(*lazy_data.values())
    [844](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:844) # evaluate all the chunked arrays simultaneously
--> [845](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:845) evaluated_data: tuple[np.ndarray[Any, Any], ...] = chunkmanager.compute(
    [846](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:846)     *lazy_data.values(), **kwargs
    [847](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:847) )
    [849](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:849) for k, data in zip(lazy_data, evaluated_data):
    [850](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/core/dataset.py:850)     self.variables[k].data = data

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:86, in DaskManager.compute(self, *data, **kwargs)
     [81](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:81) def compute(
     [82](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:82)     self, *data: Any, **kwargs: Any
     [83](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:83) ) -> tuple[np.ndarray[Any, _DType_co], ...]:
     [84](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:84)     from dask.array import compute
---> [86](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/xarray/namedarray/daskmanager.py:86)     return compute(*data, **kwargs)

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:661, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    [658](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:658)     postcomputes.append(x.__dask_postcompute__())
    [660](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:660) with shorten_traceback():
--> [661](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:661)     results = schedule(dsk, keys, **kwargs)
    [663](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/dask/base.py:663) return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:468, in _DispatcherBase._compile_for_args(self, *args, **kws)
    [464](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:464)         msg = (f"{str(e).rstrip()} \n\nThis error may have been caused "
    [465](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:465)                f"by the following argument(s):\n{args_str}\n")
    [466](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:466)         e.patch_message(msg)
--> [468](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:468)     error_rewrite(e, 'typing')
    [469](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:469) except errors.UnsupportedError as e:
    [470](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:470)     # Something unsupported is present in the user code, add help info
    [471](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:471)     error_rewrite(e, 'unsupported_error')

File ~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:409, in _DispatcherBase._compile_for_args.<locals>.error_rewrite(e, issue_type)
    [407](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:407)     raise e
    [408](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:408) else:
--> [409](https://file+.vscode-resource.vscode-cdn.net/Users/krishnaglodha/Desktop/trash/planetstac/sentin/~/anaconda3/envs/stac/lib/python3.12/site-packages/numba/core/dispatcher.py:409)     raise e.with_traceback(None)

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
('wrong tuple length for $130load_attr.11: ', 'expected 2, got 3')
During: typing of exhaust iter at /Users/krishnaglodha/anaconda3/envs/stac/lib/python3.12/site-packages/xrspatial/multispectral.py (772)

File "../../../../anaconda3/envs/stac/lib/python3.12/site-packages/xrspatial/multispectral.py", line 772:
def _normalized_ratio_cpu(arr1, arr2):
    <source elided>
    out = np.full(arr1.shape, np.nan, dtype=np.float32)
    rows, cols = arr1.shape
    ^

Expected behavior I should be able to print max value Code

catalog = pystac_client.Client.open('https://earth-search.aws.element84.com/v1')

time_range = f"{start_date}/{end_date}"
collection = "sentinel-2-l2a"
geometry = polygon["features"][0]["geometry"]
search = catalog.search(
    collections=[collection], intersects=geometry, datetime=time_range
) 
print(search.get_all_items())
data = load(
            search.items(),
            geopolygon=geometry,
            chunks={},
            resolution=1,
            projection="epsg:3857",
        )
dc = data.rio.clip([geometry], crs="epsg:4326")
index_xarray = ms.ndvi(dc.nir, dc.red)

for val in index_xarray:
    print(val.max().compute().values.item())

Desktop (please complete the following information):

Additional context Add any other context about the problem here.

brendancol commented 6 months ago

@krishnaglodha thanks for submitting the issue. I'll take a deeper look at NDVI and see if anything changed with recent 3.12 xarray-spatial release.

Can you tell post what type dc.nir.data is...and dtype, and shape?

What happens if you make sure dc.nir etc are 2d arrays by squeezing out any empty dimensions?

krishnaglodha commented 6 months ago
Screenshot 2024-05-27 at 10 30 32 AM

what baffled me is that is worked when I passed

data = load(
            search.items(),
            geopolygon=geometry,
            chunks={},
            resolution=1,
            projection="epsg:3857",
        ).isel(time=0)

i.e. for time =0

but when I try to get

data = load(
            search.items(),
            geopolygon=geometry,
            chunks={},
            resolution=1,
            projection="epsg:3857",
        )

and then loop through data , I get error

brendancol commented 6 months ago

@krishnaglodha you may be triggering a computation when running isel...I'm not totally sure.

It sounds like there may be one layer in the response that does not have the dimensionality that you expect. If you loop through the layers and print each time step out, do you see that there are only two dimensions?