spacetelescope / stcal

https://stcal.readthedocs.io/en/latest/
Other
10 stars 32 forks source link

Speed up Jump detection #229

Closed WilliamJamieson closed 11 months ago

WilliamJamieson commented 11 months ago

This PR dramatically speeds up Roman's jump detection (and ramp fitting to a lesser extent). Empirical testing locally showed me a speed up from ~33sec to ~6sec for fitting a test data set using jump detection.

The main culprit for this problem was the fact that to use numpy one has to jump out of the C code temporally. This overhead was orders of magnitude larger (relative to the expected data sizes, number of resultants per pixel in this case) than just running the computation directly in C using cython commands despite the fact that the numpy implementations of those numerical operations are much more highly optimized. Steps have also been taken to remove an unnecessary calls out of C to python which also has lead to a modest speed improvement.

~Note that there maybe additional speed improvements by explicitly aligning and indicating the contiguous array dimensions in the memory views.~

I performed this experiment but found essentially no improvements. In fact it might be slower because our expected input data is not going to be contiguous in the dimension that would result in speed improvements. In order to make this contiguous we would have to copy the input data which is quite expensive.

Checklist

codecov[bot] commented 11 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Files Coverage Δ
setup.py 0.00% <ø> (ø)
src/stcal/ramp_fitting/ols_cas22/__init__.py 100.00% <100.00%> (ø)
tests/test_jump_cas22.py 98.20% <100.00%> (-0.15%) :arrow_down:

:loudspeaker: Thoughts on this report? Let us know!