spacetelescope / stcal

https://stcal.readthedocs.io/en/latest/
Other
10 stars 32 forks source link

AL-875: Add memory saving options to compute_weight_threshold sigma_clip call #312

Closed emolter closed 1 day ago

emolter commented 1 week ago

Resolves AL-875

This PR adds memory-saving options to a call to astropy sigma clip within compute_weight_threshold in an attempt to decrease the overall memory usage of outlier detection step (see flamegraphs on JP-3685), which cause that function to modify the input array in-place instead of copying it and allocating extra arrays for masks. Since the output sigma-clipped array is immediately collapsed into its mean, there is no need to retain it as a masked array and the behavior is unchanged.

Note that the "masked=False" option to sigma_clip does not affect that function's processing of input masked arrays, it only specifies whether the output is a masked array with clipped pixels masked, or a simple np.array with clipped pixels removed.

Tasks

news fragment change types... - ``changes/.apichange.rst``: change to public API - ``changes/.bugfix.rst``: fixes an issue - ``changes/.general.rst``: infrastructure or miscellaneous change
codecov[bot] commented 1 week ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 86.35%. Comparing base (60bd3b8) to head (22252ff). Report is 6 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #312 +/- ## ========================================== + Coverage 86.21% 86.35% +0.14% ========================================== Files 47 49 +2 Lines 8812 8905 +93 ========================================== + Hits 7597 7690 +93 Misses 1215 1215 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

emolter commented 1 week ago

I'm pretty confident this will not change anything downstream, as the unit tests for the changed function appear to be relatively robust, but I started some downstream JWST regression tests here anyway

edit: these failures are unrelated