pbeaucage
commented
1 year ago Ok, not a crisis. This was mainly just renamed, BUT
recent xarray also adds map_blocks which can be parallelized in a streaming way like apply_ufunc.
https://docs.xarray.dev/en/stable/generated/xarray.map_blocks.html#xarray.map_blocks
This would let us finally use dask- or hdf5-backed arrays that don't need to hold all data in memory at once! Huzzah!
- [ ] Switch dataarray.apply calls to dataarray.map
- [x] Explore map_blocks (which is an unstable/prerelease API and we can't use anyway due to incompatibility with xarray 2022.06.0)
Warning: src\PyHyperScattering\Fitting.py:18: PendingDeprecationWarning: GroupBy.apply may be deprecated in the future. Using GroupBy.map is encouraged
Must look into this...