Improve Background Spectra Combination for NIRSpec MOS Master Background

[stscijgbot-jp]

Issue JP-3746 was created on JIRA by Christian Hayes:

We would like to add an option for median combination of background spectra (or another method that handles NaNs and outliers) in the MOS master background step.

 

Details:

In the current spec2 pipeline's NIRSpec MOS master background step, the 1D spectra extracted from all background slits are equally weighted in the combination that creates the 1D master background spectrum. Currently this can introduce bad pixels/missing CRs/etc in the final 1D master background spectrum (which has been reported by users) and can include contaminated backgrounds.  The attached image (mos_master_background_nan_badpix.png) shows an example of an automatic MOS master background spectrum (both the 1D spectrum and one of the projected 2D spectra), which shows an example of a bad pixel (at just over 3 um) and NaNs (at just under 3 um) have been propagated from individual background spectra into the MOS master background spectrum, and will be propagated to all science spectra in the exposure

Currently, the MOS master background step calls functions from combine_1d whose weight options are not exposed to the users and provide identical weights for MOS master background spectra (the weighting options are "integration_time," "exposure_time," and "unit_weight", all of which would be the identical for background slitlets in the same exposure).  We would like to have alternative options (exposed to the users) for combining spectra, preferably some that include a manner of outlier rejection, such as a median or variance-weighted combination of the input spectra.  Particularly a median combination may remove most cases of bad or missing pixels and improve the default processing for this step. 

For example the two plots mos_mean_background.png and mos_median_background.png show examples of MOS background spectra and compare a mean or median 1D spectrum (with NaN handling), showing that in addition to rejecting bad pixels, a median combination will also in principle filter out bad background spectra (e.g., spectra that are contaminated or may come from partially stuck closed shutters).

[stscijgbot-jp]

Comment by David Law on JIRA:

Assigning this to NIRSpec while we sort out the best approach.  Christian Hayes I think there are some common themes between this and potential MRS background improvements too that we can discuss.

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

David Law who would be best to talk to about the MRS background improvements that MIRI wants?

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

I have put together a potential solution to improve the MOS background combination here:  https://github.com/hayescr/jwst/tree/improve_combine1d  and we are looking into it on the NIRSpec team.  

This adds the option to provide a "sigma_clip" parameter to combine_1d_spectra in combine_1d.  If a "sigma_clip" value is passed, the code will take all of the input spectra, calculate a pixel-wise median spectrum and a median absolute deviation and then clip any pixels in the input spectra that are more than sigma_clip * MAD away from the median.  It will then continue to do the weighted sum as the algorithm currently does (though I've refactored this to be able to modify the input spectra before summing them and to handle nans that were slipping through).

Doing some initial testing I've found that this seems to do fairly well at rejecting outliers (both contaminated spectra and bad pixels).  I've attached an example ([^masterbkg_comparison_jw01448011001_02101_00001_nrs2.pdf]) showing the same dataset as originally added to the ticket.  The top panels show the old MOS master bacgkround and the bottom show products from this branch.  Left shows the masterbg2d for one of the slits, and the right shows the masterbg1d spectrum compared to all of the input background spectra.  

This is an extreme case with > 100 background spectra, but it also appears to work well for a couple examples I tested with fewer backgrounds in grating observations (e.g., PID 1810 obs 1, jw01810001001_09101_00004_nrs2 is particularly a good example with only 10 background slits defined some which are contaminated).

We are still looking into this a bit more to see if it looks like it will work more generally for NIRSpec/MOS observations, but I wanted to share the update in case this is along the lines of what MIRI was interested in for MRS. 

[stscijgbot-jp]

Comment by David Law on JIRA:

Thanks Christian Hayes ; I'll test this out for MRS too.  One thing that we'd been considering for MRS was introducing some degree of spectral smoothing in the master background, potentially by using some kind of spline model, or even just a moving boxcar.  This is motivated by our short-wave data in particular, where we're really dominated by detector artifacts more than any real background signal, but it looks like it may also be useful for NIRSpec as the background doesn't have sharp spectral features.

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

David Law agreed, we had been thinking about whether we could include a smoothing option as well (to smooth the final background spectrum).  Especially when there are only a few backgrounds, they can either be quite noisy or even include a couple remaining detector artifacts, either of which might be nice to smooth over.  When I get a chance I can take a bit of a look and see if there might be some useful smoothing options, but we'd also be happy to chat with people on the MIRI team to see what you all were thinking/had tried.

[stscijgbot-jp]

Comment by David Law on JIRA:

Christian Hayes I don't think the MOS improvements in your branch above will help out much for MRS, largely because we're typically using just 1-2 input background spectra so the sigma clipping doesn't have enough to work on.

That said, I've added a moving-boxcar median filtering now as well, with a PR against your repo at https://github.com/hayescr/jwst/pull/2/files

I kept going back and forth on whether to add this in combine1d like your changes, but ultimately opted to keep it in the master_background_step code since it's really an operation performed on the 1d-combined spectrum rather than a different way to create a combined spectrum.  You'd thus have to implement something similar like it for master_background_mos_step if you want it to be available for MOS as well.

Note also that this explicitly doesn't look at the variance; it ONLY median filters the surf_bright array that's used for the master background computation.  Variance of the median filter isn't terribly well defined, so I'm not trying to change all of the various arrays returned by combine1d.

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

Thanks, David Law, we are also considering a median filtering and possibly leaving anything more sophisticated for users to do.  I agree that it probably makes more sense to have this in master_background rather than combine1d.  I left a couple small questions for you on your PR to my branch, but I'm happy to make any necessary changes.

[stscijgbot-jp]

Comment by Christian Hayes on JIRA:

Tyler Pauly Nadia Dencheva I'm not sure who would be best to take a look at this, but we have a draft PR for this ticket here:  https://github.com/spacetelescope/jwst/pull/8932].  I'm keeping it in draft for the moment while I finish updating some of the tests and documentation, but wanted to give you all a head's up that this will be ready for review soon.