TUDelftGeodesy / stmtools

Xarray extension for Space-Time Matrix
https://tudelftgeodesy.github.io/stmtools/
Apache License 2.0
7 stars 0 forks source link

Performance warning in re-order #73

Closed rogerkuou closed 4 months ago

rogerkuou commented 6 months ago

The demo notebook or reordering gives performance warning when calling re-order:

# Time the reordering operation.
time_ordering = %timeit -o stmat.copy().stm.reorder(xlabel="azimuth", ylabel="range")
time_ordering
/storage/miniforge3/envs/mbl_stmtools/lib/python3.11/site-packages/xarray/core/indexing.py:1430: PerformanceWarning: Slicing with an out-of-order index is generating 230 times more chunks
  return self.array[key]

This seem to be inevitable since we are performing reorder. It seems we can

  1. Supress the warning
  2. Check some Dask reordering, e.g. :https://docs.dask.org/en/stable/order.html
vanlankveldthijs commented 4 months ago

As we discussed, this performance would warning probably only apply while the data is split into a lot of small chunks (in this case 230 times as many).

However, inside the stm.reorder(...) function, the data is already re-chunked into larger chunks directly after this call to xarray.Dataset.sortby(...).

So, I will suppress the warning and check whether the function description properly mentions that this could be an expensive operation.