search

spacetelescope / jwst

Python library for science observations from the James Webb Space Telescope
https://jwst-pipeline.readthedocs.io/en/latest/
Other
570 stars 167 forks source link

`test_miri_mrs_badpix_selfcal_bkg` fails okify #8968

Closed braingram closed 2 days ago

braingram commented 2 days ago

While okifying regression tests results I noticed that test_miri_mrs_badpix_selfcal_bkg can require okifying several times due to the structure of the test: https://github.com/spacetelescope/jwst/blob/57f6148f16e49025927d76a3b61d351b2c49f0ae/jwst/regtest/test_miri_mrs_badpix_selfcal.py#L71-L80 The test has multiple comparisons between result and truth files. This is incompatible with the "postmortem" which generates an upload schema detailing which files to replace. There are likely some details I'm missing here so more investigation will be needed but I believe what's happening is as follows.

Let's assume a change produces different results for all 5 comparisons (as was the case for https://github.com/spacetelescope/stdatamodels/pull/354). The first regtest run and okify will only replace the truth file for the first comparison (since the later truth files were never fetched and "rtdata.truth" and "rtdata.truth_remote" added to the "upload schema" refer to the first, failed comparison). This will mean the second time regtests are run the first comparison will succeed and the second will fail (etc...). So it's possible this test will need to be okified 5 times.

Perhaps investigating a way to enforce "one truth per test" is worthwhile since that's what "rtdata" seems to expect.

braingram commented 2 days ago

This test will always fail to okify since it doesn't set rtdata.output for the background comparisons.