search

spacetelescope / stpipe

https://stpipe.readthedocs.io
Other
3 stars 25 forks source link

Add `ModelLibrary` container class #156

Closed braingram closed 2 months ago

braingram commented 3 months ago

This PR adds a new container class ModelLibrary to be used in both the jwst and romancal pipelines.

The main benefits of the new class are covered in the docs added in this PR

The "tldr" is that this container class can either use "in memory" or "on disk" models allowing memory saving for many pipeline steps and is compatible with the step and pipeline machinery allowing on disk containers to pass between steps in a pipeline (something that isn't possible with ModelContainer).

Although final PRs for jwst and romancal are not yet ready (and will depend on the final merged version of this PR) branches for both pipelines are available for review as part of this PR.

The jwst branch https://github.com/spacetelescope/jwst/compare/master...braingram:jwst:stpipe_library focuses only on updating steps involved in calwebb_image3 at the moment (and requires other ongoing work refactoring outlier detection before the model library integration can proceed).

The romancal branch has a draft PR (that is at the moment out-of-date) https://github.com/spacetelescope/romancal/pull/1241 The focus for the romancal work is to fully replace ModelContainer.

Note that the below regression tests use the spacetelescope forks and default branches for jwst and romancal (not the above linked branches which aren't yet ready for review). Roman regression tests run: https://github.com/spacetelescope/RegressionTests/actions/runs/9861615006 shows only unrelated failures that also occur on main.

JWST regression tests run with no errors at: https://plwishmaster.stsci.edu:8081/job/RT/job/JWST-Developers-Pull-Requests/1594/

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is 90.63545% with 28 lines in your changes missing coverage. Please review.

Project coverage is 73.20%. Comparing base (e074807) to head (89e8262). Report is 4 commits behind head on main.

Files Patch % Lines
src/stpipe/step.py 25.00% 21 Missing :warning:
src/stpipe/pipeline.py 16.66% 5 Missing :warning:
src/stpipe/library.py 99.24% 2 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #156 +/- ## ========================================== + Coverage 69.83% 73.20% +3.36% ========================================== Files 24 25 +1 Lines 1618 1911 +293 ========================================== + Hits 1130 1399 +269 - Misses 488 512 +24 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.