NIRSpec BOTS data need to be rectified

[stscijgbot-jp]

Issue JP-1488 was created on JIRA by James Muzerolle:

The resample_spec step needs to be run on BOTS data before 1D spectra are extracted.  Currently, the calwebb_tso-spec2 and _tso3 pipelines skip this step, which leads to incorrect 1D spectral products because the extract_1d algorithm is not written to account for the spectral curvature  (particularly pronounced for the gratings) and the slit tilt (wavelength not aligned to detector columns).  Since the primary BOTS data product will be the level 3 1D spectra, the fix should prioritize the tso3 pipeline.

This is critical priority for both Commissioning and science data because we need usable data products for both. There is no easy workaround because the step is hardcoded to skip BOTS data.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Implementation will require several changes:

The calwebb_tso-spec2.cfg configuration file currently has the resample_spec step set to be skipped. This will need to be removed to allow for BOTS mode (NRS_BRIGHTOBJ) to have resample_spec applied. Any remaining logic necessary to skip the step for other modes TSO modes (e.g. MIR_LRS-SLITLESS, NRC_TSGRISM, and NIS_SOSS) will need to be incorporated into the Spec2Pipeline module itself.

The Spec2Pipeline module is currently setup to only apply the resample_spec step if EXP_TYPE is NRS_FIXEDSLIT, NRS_MSASPEC, or MIR_LRS-FIXEDSLIT. NRS_BRIGHTOBJ will need to be added to this list.

The Spec2Pipeline module is currently setup to skip the resample_spec step if the data are in the form of a CubeModel, which is the case for NRS_BRIGHTOBJ. So this logic will need to be modified to allow resample_spec to be called for NRS_BRIGHTOBJ CubeModel data.

The resample_spec step itself is not currently setup to allow for processing of 3-D data (CubeModel); it will only handle 2-D images. Hence a workaround needs to be implemented to either temporarily reformat the 3-D as a ModelContainer with multiple 2-D models, or modify the resample_spec step to allow for looping over the planes of a 3-D CubeModel. We don't want the multiple planes combined, as is done with 3-D coronagraphy data, but instead each plane needs to be resampled individually.

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

James Muzerolle It occurred to me while creating the list of needed changes in the above comment, that the work is focusing completely on calwebb_tso-spec2 processing. But in this pipeline, the resampled (s2d) and extracted (x1d) products are simply dead-end, quick-look products for each exposure and are not the final pipeline product for a TSO observation. The final products come from calwebb_tso3 processing, which includes outlier_detection, extract_1d, and white_light steps. Note that this level 3 pipeline doesn't currently include resampling at all (for any mode). So instead of making several mods to enable calwebb_tso-spec2 to produce resampled products, would it be better to concentrate the work on adding resample_spec to the calwebb_tso3 pipeline, for only NRS_BRIGHTOBJ data? The final extracted spectra (x1d product) would then have the benefit of outlier_detection and resampling.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Howard Bushouse yes, you're absolutely right.  I was fixated on spec2 because I had just been testing it on other modes.  I'll edit the description to reflect this.

In the meantime, perhaps we should turn off extract_1d in tso_spec2, just to avoid any confusion as to whether those intermediate products are usable.

[stscijgbot-jp]

Comment by Jeff Valenti on JIRA:

Before changing the TSO pipeline specifications, let's get our exoplanet stakeholders to comment in this ticket.

Nestor Espinoza and Sarah Kendrew. Please comment in this ticket whether you as TSO WG leads endorse resampling pixels before extracting a 1D spectrum. Isn't conventional wisdom for ultra-high precision TSO data that resampling should be avoided because it couples errors in neighboring spectrum points? Do we expect users to do ultra-precise science with extracted 1D spectra or only 2D spectra?

James Muzerolle. Please elaborate on your statement that the extract_1d algorithm does not account for spectral curvature, perhaps by pointing me at the specification for extract_1d. I'm not asking about spectral tilt, which I understand is difficult to handle without resampling. 

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

I agree with your assessment Jeff Valenti that rectification of the spectrum is typically not desired, and what is typically done in curved spectra is to perform the spectral extraction using the 2D wavelength solution (see, e.g., the case for HST in Section 2.5 here: https://iopscience.iop.org/article/10.3847/0004-637X/832/2/202]). It is of my understanding, however, and based on an e-mail by James Muzerolle not related to this ticket, that rectification is done in all the other NIRSpec modes and I presume this is where the issue lies (because the team has already delivered algorithms with that step in mind). I see, thus, three ways forward:

1. Add the rectification step, so the rest of the process works for the NIRSpec team (less work, but probably will involve systematics in the Level 3 products that users might want to become aware of). Make TSO users aware (e.g., via JDox) that this is the only mode for which spectral rectification is done prior to 1D-extraction for TSOs. 

2. Don't add the rectification step, but add an extraction algorithm that accounts for the fact that the spectrum is tilted. This is needed because, as stated in this ticket, the vanilla extract1d algorithm simply adds flux on each column, which will also imply systematics at the Level 3 products. This is, of course, more work, as the NIRSpec team would have to deliver this before the end of the month if they would want it included in version 7.6 of the JWST pipeline.

3. Don't add anything (yet), but make sure users are aware of the problems with the 1D-extracted spectra and, thus, the Level 3 products.

I think that given both the unrectified and rectified 2D products are given by the pipeline ([https://jwst-pipeline.readthedocs.io/en/latest/jwst/pipeline/calwebb_spec2.html#calwebb-spec2]), perhaps adding the rectification (i.e., option 1 above) would not be too bad for now (i.e., to be included in the flight build), but the optimal would be option number 2 above.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Jeff Valenti The default BOTS processing collapses along columns over the full cross-dispersion width of the spectral trace, with the extraction aperture centered in the middle.  This currently produces 1D spectra that are of limited value (at least for the gratings), even by quick-look standards.  There are several options to account for the curvature in this framework without resampling: 1) we can specify a polynomial to fit the curved trace in the extract_1d reference file; 2) we could have the code find the center of the trace either by using the WCS or by fitting the spatial profile.  Option (1) is the easiest, and would require no code changes.  However, neither of these options would account for the spectral tilt.  Since we already have spectral resampling for the other modes, I thought that would be the best short-term solution to provide reasonable if non-optimal 1D data products.

I think Nestor sums up the different options well, and agree that his option (2) would be the optimal approach.  That requires a new algorithm, which would not be ready in time for inclusion in b7.6.

[stscijgbot-jp]

Comment by Jeff Valenti on JIRA:

Thanks Nestor Espinoza and James Muzerolle for the helpful comments. What is the scientific impact if we address curvature of the trace (e.g., one of the options James described in the previous comment), but do not address tilt of the spectral lines? I assume spectral resolving power is degraded, but is that really a problem for TSO work where observers often bin wavelengths anyway? If necessary, users could convolve their model spectrum with asymmetric LSF functions that we provide. Are there other impacts of ignoring spectral line tilt for TSO extractions?

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

It depends on how strong the spectral tilt is. The problem is not really on the resolving power, but on signals that might leak to neighboring wavelength bins (say, abrupt absorption features that might only appear in a couple of pixels). Having light from neighboring wavelengths leaking into your spectral feature would impact their shape (because it will get smeared out) and strength (because you'll have slight dilution from neighboring pixels) — not to mention its impact in, e.g., limb-darkening. James Muzerolle might comment on how strong the tilt is — based on that we could study how bad it would be not to include it in the extraction.

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

For the gratings, the tilt is 2-3 degrees, varying slightly with wavelength.  This translates to lines of constant wavelength crossing ~2-3 pixels over the entire ~15 pixel height of the S1600A1 slit, and so will have less of an effect over the source itself.  There may be a larger impact on background subtraction, depending on how the background region is selected.

The tilt is larger for PRISM, around 15 degrees.  But of course, narrow lines won't be of interest in that case.

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

Hi James Muzerolle --- if that's the case, then I don't think this would outweigh the fact that rectification/resampling would produce much bigger issues on further steps of the reduction. For instance, I'm unsure if the previous steps in the pipeline are able to properly remove the 1/f noise that appears on a column-by-column basis in NIRspec frames (from experience with other instrument modes, it is highly doubtful this is properly removed in Step 1 and in all the steps previous to this one in Step 2 of the pipeline). If not accounted for, this would give rise to very large systematics in the final time-series, which I believe would be far worse than having some contamination from neighboring pixels in the worst-case scenario in which extract1D simply performs column-by-column sums. Performing rectification of the 2D image would, for instance, probably de-rectify the 1/f noise components, making its removal in later steps much more complicated.

My opinion on this matter then is to not apply rectification as in the other TSO modes. It should be noted, however, that I think a proper 1d extraction algorithm should be eventually written for this mode in order to account for the spectral tilt (which although it is small, it could be important for some applications).

[stscijgbot-jp]

Comment by James Muzerolle on JIRA:

Given the consensus that rectification would introduce more systematics compared to extracting unrectified data, I'm withdrawing this ticket.  A new ticket will be submitted in the future once an updated algorithm has been designed to deal with the spectral curvature & tilt.