search

usnistgov / PyHyperScattering

Tools for hyperspectral x-ray and neutron scattering data loading, reduction, slicing, and visualization.
Other
6 stars 8 forks source link

Dask-backed reduction #50

Closed pbeaucage closed 1 year ago

pbeaucage commented 1 year ago

adding dask backed reduction

pbeaucage commented 1 year ago

This is in a state where I'd consider merging.

Status:

Dask reduction is working for warp_polar and for pyFAI with the exception of reduction with a multiindex; if you have a multiindex there is a weird failure mode that I think might be upstream. The good news is that Dask basically gets rid of the need for multi indices in the first place.

There are simple flags use_chunked_loading = True to the constructor of cyrsoxsLoader and SST1RSoXSDB. Doing this makes them return Dask arrays, which can be easily massed to the integrators, which take a similar constructor argument use_chunked_processing = True. This pipeline works and is fast as heck.

The thinking is this:

getting this into the pre-release stream is going to accomplish these without, hopefully, derailing too much work.