Investigate scaling issues of simulated spectra in wfss_contam step

[stscijgbot-jp]

Issue JP-2075 was created on JIRA by Howard Bushouse:

Testing of the prototype wfss_contam step using the latest NIRCam WFSS MIRAGE simulations shows a difference in the overall signal level of simulated contamination spectra generated by the step relative to the spectra in the grism image being corrected. Check for proper application of flux calibration terms and possible differences in flux calibration data that were used in MIRAGE vs the current CRDS ref files.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Temporarily setting to on hold, until WFSS INS group can also take a look into scaling of simulated data for both NIRISS and NIRCam as part of the overall testing of the wfss_contam step.

[stscijgbot-jp]

Comment by Swara Ravindranath on JIRA:

Checked the WFSS contamination correction in the development version of the pipeline:

pipeline: 1.2.4.dev29+g7d28ed89 Some examples from the checks performed are in the attachments.

The pipeline seems to be doing the wfss contamination step correctly and in cases where the contamination from higher orders are removed the spectra are clean with only information about the source of interest. However, there are some issues with the contamination models.

(1) There is an issue with the intensity scaling in the models which causes over-subtraction of the contaminated pixels.

(2) For NIRISS WFSS, the zero order images are bright and are a cause for serious contamination. Looks like the 0th order is currently not being included in the contamination model. So, they still remain in the output image 2D-cutouts after processing.

Note: The issue of intensity scaling and not including 0th order (sometimes extremely faint because of wrong flux scaling) is something I also noted independently in my recent Mirage simulations (outside of the pipeline code).

[stscijgbot-jp]

Comment by Swara Ravindranath on JIRA:

I checked the log and the zero order is not being included in the contamination model. The issue that was seen in the MIrage simulations has been resolved and is not related to the 2 issues reported in the previous comments.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

The reason why the zero-order signal is not included in the simulated contamination model is because all of the simulated data are first created in flux units (that's just the way they come out of the models) and then require conversion back into the DN/s units of the actual science data. This is done by applying the inverse of the calibrations stored in the photom reference file and there's no calibration info for the zero-order. So the code currently avoids including the zero-order in the simulated data to begin with.

[stscijgbot-jp]

Comment by Swara Ravindranath on JIRA:

Nadia Dencheva  Howard Bushouse

Wondering whether the flux discrepancies seen for the NIRISS WFSS contamination model may be related to the issue of quadratic trace function mentioned in JP-2375. The Mirage simulations used for testing the WFSS contamination were created using the V3 NIRISS CONF files which incorporate the quadratic trace polynomials. However, the WFSS contamination step in the pipeline is using the information from reference files created based on an earlier version of the CONF file that has linear trace. If this is the cause of the problem, then NIRCam WFSS contamination step should not show the flux discrepancy because the trace is linear in that case. Do we know if NIRCam has the flux mismatch?

[stscijgbot-jp]

Comment by David Law on JIRA:

Doesn't look like there's been any progress on this since getting real data instead.  Closing as not a priority any longer unless someone feels it needs to be reopened.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

David Law This investigation has been taking place on and off over the past several months. It is not being actively worked by Ned Molter

 

edit: it is now being worked by Ned Molter

[stscijgbot-jp]

Comment by David Law on JIRA:

Roger, thanks for clarifying.

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

This was caused by a failure to account for the effect of the wavelength oversampling on the output model flux in dispersed_pixel(). The oversampling was set to 2 by default, which meant that each calculation was being double-counted, and the model was exactly twice as bright as the data.  This is fixed by simply dividing the output fluxes in that function by the oversample factor.  (edit: see figures attached to this ticket, one before dividing by oversample_factor, and another after dividing by oversample_factor)

This was made much more difficult to detect because the background of the direct image was not subtracted properly.  The (outlier-removed) mean of the data was substantially above the noise level, and this extra, unphysical flux was getting propagated into modeling the dispersion.  Source detection estimates its own background, so the segmentation map looked normal, even though the input fluxes from the map were wrong. 

Do you think we should include a simple background subtraction on the direct image within this step? I am leaning toward yes, because it's very computationally inexpensive and the step will not produce reasonable results otherwise.  If not, can anyone comment on what might be causing the issue with the direct image so we can ensure that gets fixed?

Note that I didn't re-run the calibration steps on the direct image, but the data are less than a year old and were run with pipeline version 1.13.3.  The filename of this direct image is jw02738-o001_t001_niriss_clear-f200w_i2d.fits.  

-I am planning to include this fix alongside some performance updates to the step as requested by JP-2546.  Let me know if there is a strong reason to separate these PRs and I will go ahead and submit a PR for just this change.-

Just submitted a PR for this fix only.  Let me know what you think.

[stscijgbot-jp]

Comment by David Law on JIRA:

Flagging this to the NIRISS team to provide input.  Jo Taylor Rachel Plesha 

[stscijgbot-jp]

Comment by Norbert Pirzkal on JIRA:

Just to pipe in, I can confirm that when propagating the direct imaging flux to generated 2D wfss simulated data, the background needs to be removed from the imaging data. 

[stscijgbot-jp]

Comment by Ned Molter on JIRA:

Howard Bushouse per our discussion today, I'm attaching the result of the nircam regression test.  The simulation is almost exactly a factor of 2 smaller everywhere.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Fixed by #8416

[stscijgbot-jp]

Comment by Rachel Plesha on JIRA:

The NIRISS WFSS team does not currently have the resources to test this fix. We will come back to this when we have a more solid plan for WFSS contamination modeling testing for NIRISS.