Updates and fixes for WFSS contamination correction

[stscijgbot-jp]

Issue JP-2546 was created on JIRA by Howard Bushouse:

Wide-Field Slitless Spectroscopy (WFSS) contamination correction needs some remaining work to make it fully functional, including updates to allow the use of the latest trace configuration data, proper handling of zero-order light, and unexplained offsets of signal amplitude between actual spectral sources and the simulated spectra used for contamination correction.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

Adding some known issues here that should be looked into. Strikethrough means completed; see also the more recent comment about items requiring input from instrument teams.

• -Current implementation of multiprocessing does not work.-  works now but see other comment
• -Multiprocessing may be more effective looping over segmentation map sources, instead of individual pixels within each segmentation map source.-  resolution: did not end up making sense to move this
• -Each segmentation map source is currently modeled twice, once to create simulated grism map and again to subtract each source from that map to create the contamination fields. These are not very large and could easily be stored in memory to avoid re-computing.- fixed.
• -It may be possible to ignore sources near the edge of the direct image before modeling them, instead of modeling them and then throwing out results that lie outside the FOV.  This would modestly improve runtime.- probably more annoying than it's worth; should focus elsewhere to speed up runtime
• -It should be possible to run the step using only the brightest N sources in the segmentation map.  This would make testing the step easier, because extract_2d selects for the brightest N sources.-  done. Many of the fainter sources identified in the direct image lie far below the noise level in the grism map, so this could even be investigated as a method to reduce runtime, depending on how big a difference including these faint contamination sources makes for the science case of interest.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

I am just now submitting a draft PR for this.  There are still plenty of outstanding issues to resolve, but most of these would benefit from input from other people at this stage.

Changes:

• Made multiprocessing work
• Removed necessity to simulate Order 1 of all sources twice, instead storing these in memory.  This decreases runtime.
• Added a new optional output file, _simulated_slits.fits, that stores a MultiSlitModel containing contamination-free modeled slits for all requested spectral orders and sources.  I found this very useful for examining the step's outputs, but a decision should be made about whether this functionality is desired in the final version of the step.
• Added brightest_n parameter to specify how many sources to simulate.  This facilitates testing by decreasing runtime.
• Added unit testing suite with coverage for most helper functions.
• Did some refactoring to enhance readability and ease of unit testing.

Outstanding issues:

• Output needs to be examined by the instrument teams for additional outstanding issues.
• Zeroth order contamination for NIRISS needs to be calibrated.
• Decide whether the new output file is actually desired.
• Loading sources from cache used to be supported, but I have not turned this mode on, nor tested it.  A decision should be made about whether this functionality is desired.  If so, some updates are required to support this, and if not, all related functions need to be removed.
• Using multiple direct images in different filters to give basic spectral information to model spectra is supported in principle, but currently untested. If this functionality is desired, a test dataset where this mode is being used would be extremely helpful.
• One regression test each should be written for NIRCam and NIRSS, ideally using a dataset and result that has been vetted by the instrument teams.

Potential longer-term improvements that I don't think should go in this PR:

• It may be possible to ignore sources near the edge of the direct image before modeling them, instead of modeling them and then throwing out results that lie outside the FOV.  This would modestly improve runtime.
• Support for inputting assumed/previously-known source spectra to improve grism model.

I am curious to hear what else y'all can think of that should still be done.

[stscijgbot-jp]

Comment by Jo Taylor on JIRA:

Thanks for the progress Ned! We have a NIRISS WFSS meeting in a couple of weeks and will discuss the changes here at that time.

[stscijgbot-jp]

Comment by David Law on JIRA:

Bumping this ticket to see whether Ned Molter still needs feedback from the INS teams.

[stscijgbot-jp]

Comment by Rachel Plesha on JIRA:

The NIRISS WFSS team does not currently have the resources to test the contamination modeling. We will come back to this when we have a more solid plan for WFSS contamination modeling testing for NIRISS. This is still a high priority, there are just other higher priorities we need to look into first.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

From the pipeline team side, yes David Law that is the status - on hold for now.

No worries Rachel Plesha, I know this takes a lot of work on your end. 

Just to note, I do think a few of the questions listed above could be answered without testing the step in detail or doing really anything with my PR branch, namely:

• Decide whether the new output file is actually desired.
• Loading sources from cache used to be supported, but I have not turned this mode on, nor tested it.  A decision should be made about whether this functionality is desired.  If so, some updates are required to support this, and if not, all related functions need to be removed.
• Using multiple direct images in different filters to give basic spectral information to model spectra is supported in principle, but currently untested. If this functionality is desired, a test dataset where this mode is being used would be extremely helpful.

If these points were decided, then the step I eventually deliver to you for preliminary testing would be that much closer to the final product.  I know deciding these still takes substantial effort for the WFSS team, but maybe less effort than a full test of the step.