search

xarray-contrib / flox

Fast & furious GroupBy operations for dask.array
https://flox.readthedocs.io
Apache License 2.0
123 stars 16 forks source link

Vectorized mode #391

Closed dcherian closed 3 weeks ago

dcherian commented 3 weeks ago

This is some WIP work, that it would be good to take over the finish line

cc @BSchilperoort

BSchilperoort commented 3 weeks ago

To illustrate what I want to do with the mode aggregation I created this gist: https://gist.github.com/BSchilperoort/67ccfbb7f6d0e0b9a8df069c41a5db24

You can run it if you have xarray-regrid, zarr, and gcsfs installed. The notebook runs on the latest release of flox. This method works, but gets very memory hungry with input data at a finer resolution.

If I check out this branch, and rerun the notebook, the reduction fails with the following error (but I guess this PR is still very WIP?):

Details

```py File ~/micromamba/envs/xarray_regrid_3.12/lib/python3.12/site-packages/flox/aggregations.py:147, in _atleast_1d(inp, min_length) 145 if xrutils.is_scalar(inp): 146 inp = (inp,) * min_length --> 147 assert len(inp) >= min_length 148 return inp ```