JP-2928: Update MIRI MRS test data

[melanieclarke]

Partially addresses JP-2928

Update regression test data to in-flight examples for MIRI TSO and master background correction with a dedicated background observation.

For TSO, the data is from PID 1556 obs 1, detector = mirifushort.

For the background correction, the input data is a subset of the files in jw01031-c1004_20241028t205539_spec3_00001_asn.json, from PID 1031, obs 1 (science) and obs 6 (background). I included only the mirifulong inputs for the first sequence of both science and background observations. The input includes 4 cal files and 2 background spectra, equivalent to the (simulated) inputs for the existing test.

Tasks

• [x] request a review from someone specific, to avoid making the maintainers review every PR
• [x] add a build milestone, i.e. Build 11.3 (use the latest build if not sure)
• [x] Does this PR change user-facing code / API? (if not, label with no-changelog-entry-needed)
  • [ ] write news fragment(s) in changes/: echo "changed something" > changes/<PR#>.<changetype>.rst (see below for change types)
  • [x] update or add relevant tests
  • [ ] update relevant docstrings and / or docs/ page
  • [x] start a regression test and include a link to the running job (click here for instructions)
  • [x] Do truth files need to be updated ("okified")?
    • [ ] after the reviewer has approved these changes, run okify_regtests to update the truth files
• [ ] if a JIRA ticket exists, make sure it is resolved properly

news fragment change types...

- ``changes/.general.rst``: infrastructure or miscellaneous change - ``changes/.docs.rst`` - ``changes/.stpipe.rst`` - ``changes/.datamodels.rst`` - ``changes/.scripts.rst`` - ``changes/.fits_generator.rst`` - ``changes/.set_telescope_pointing.rst`` - ``changes/.pipeline.rst`` ## stage 1 - ``changes/.group_scale.rst`` - ``changes/.dq_init.rst`` - ``changes/.emicorr.rst`` - ``changes/.saturation.rst`` - ``changes/.ipc.rst`` - ``changes/.firstframe.rst`` - ``changes/.lastframe.rst`` - ``changes/.reset.rst`` - ``changes/.superbias.rst`` - ``changes/.refpix.rst`` - ``changes/.linearity.rst`` - ``changes/.rscd.rst`` - ``changes/.persistence.rst`` - ``changes/.dark_current.rst`` - ``changes/.charge_migration.rst`` - ``changes/.jump.rst`` - ``changes/.clean_flicker_noise.rst`` - ``changes/.ramp_fitting.rst`` - ``changes/.gain_scale.rst`` ## stage 2 - ``changes/.assign_wcs.rst`` - ``changes/.badpix_selfcal.rst`` - ``changes/.msaflagopen.rst`` - ``changes/.nsclean.rst`` - ``changes/.imprint.rst`` - ``changes/.background.rst`` - ``changes/.extract_2d.rst`` - ``changes/.master_background.rst`` - ``changes/.wavecorr.rst`` - ``changes/.srctype.rst`` - ``changes/.straylight.rst`` - ``changes/.wfss_contam.rst`` - ``changes/.flatfield.rst`` - ``changes/.fringe.rst`` - ``changes/.pathloss.rst`` - ``changes/.barshadow.rst`` - ``changes/.photom.rst`` - ``changes/.pixel_replace.rst`` - ``changes/.resample_spec.rst`` - ``changes/.residual_fringe.rst`` - ``changes/.cube_build.rst`` - ``changes/.extract_1d.rst`` - ``changes/.resample.rst`` ## stage 3 - ``changes/.assign_mtwcs.rst`` - ``changes/.mrs_imatch.rst`` - ``changes/.tweakreg.rst`` - ``changes/.skymatch.rst`` - ``changes/.exp_to_source.rst`` - ``changes/.outlier_detection.rst`` - ``changes/.tso_photometry.rst`` - ``changes/.stack_refs.rst`` - ``changes/.align_refs.rst`` - ``changes/.klip.rst`` - ``changes/.spectral_leak.rst`` - ``changes/.source_catalog.rst`` - ``changes/.combine_1d.rst`` - ``changes/.ami.rst`` ## other - ``changes/.wfs_combine.rst`` - ``changes/.white_light.rst`` - ``changes/.cube_skymatch.rst`` - ``changes/.engdb_tools.rst`` - ``changes/.guider_cds.rst``

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 63.68%. Comparing base (1f87cb2) to head (3cb6359). Report is 23 commits behind head on main.

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #8936 +/- ## ======================================= Coverage 63.68% 63.68% ======================================= Files 375 375 Lines 38689 38691 +2 ======================================= + Hits 24638 24640 +2 Misses 14051 14051 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[melanieclarke]

Regression tests here: https://github.com/spacetelescope/RegressionTests/actions/runs/11615060117

One unrelated failure, also failing on main.

[melanieclarke]

Thanks to feedback from @drlaw1558 and @jemorrison, I have removed the test for master background correction for nodded MIRI MRS data (test_miri_mrs_nod_masterbg.py). Master background corrections only proceed if there is a dedicated background observation, so test_miri_mrs_dedicated_mbkg should cover that case.

[jemorrison]

I wanted to review the regression tests results since this is a new system to me and I wanted to see how they looked. when I click on the link in this pr to regression tests it just shows a nirspec failure - is that because there are no failures for these new tests because you updated the truth files

[melanieclarke]

I wanted to review the regression tests results since this is a new system to me and I wanted to see how they looked. when I click on the link in this pr to regression tests it just shows a nirspec failure - is that because there are no failures for these new tests because you updated the truth files

Yes, that's right. I generated the truth files with this branch on my local system and uploaded to artifactory before running the regression tests. They don't overlap with the old versions, so there are no conflicts with the truth files for the tests on main. The point in running them that way is mostly to make sure there aren't any architectural differences or installation problems in my local version versus the regression test system, as well as to make sure I haven't accidentally broken anything else.