spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
568 stars 167 forks source link

memory and runtime benchmark/testing infrastructure #8858

Open stscijgbot-jp opened 1 month ago

stscijgbot-jp commented 1 month ago

Issue JP-3775 was created on JIRA by Ned Molter:

Add and implement tools to ensure that pull requests do not inadvertently cause significant regressions in memory usage or runtime.

At a hack day discussion, we targeted initial implementation and testing in stcal, since it's a smaller repository but still has non-trivial memory- and runtime-hungry functions (e.g., outlier detection median calculation).  Goal is very quick adoption by JWST, and adoption by Roman once other outstanding regression test issues have been resolved.

The preferred workflow would be to have decorators that could be added to tests to flag them as memory- and/or runtime-checked.  At time of writing, codspeed is being considered for runtime, and pytest-memray is being considered for memory.  

Task list for now:

stscijgbot-jp commented 1 month ago

Comment by Ned Molter on JIRA:

A PR in stcal adds a context manager that makes it easier to compare the peak memory usage of a code block against an expected usage.  A single unit test to the median calculator is implemented that uses the context manager, and this test should be used as an example of how to implement a test where the expected memory usage can be calculated straightforwardly based on the input size.  The new context manager can be imported into roman and jwst.

PR link here: spacetelescope/stcal#299 

edit: this part of the work is complete