Spec2Pipeline WFSS DQ flags are not correctly propagated to extract_1d step

[stscijgbot-jp]

Issue JP-1921 was created on JIRA by Swara Ravindranath:

In Spec2 pipeline for NIRISS WFSS the DQ flags are not propagated correctly to the extract_1d step and the bad pixels are not excluded when pixels are summed in a given wavelength bin.

Issue was noted in the combined output product and reported in JP-1621. This new ticket was opened after verifying that the issue was not with the combine_1d step in Spec3 pipeline.

[stscijgbot-jp]

Comment by Swara Ravindranath on JIRA:

The figures show the combined spectrum from Spec3 pipeline, and the 4 extracted 1_d spectra from the Spec2 pipeline corresponding to 4 dithers that are combined in Spec3.

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

We had a brainstorming session for this in the DMSWG: https://outerspace.stsci.edu/display/JWSTPWG/2021-02-17+Meeting+notes%3A+DMSWG

The teams need to agree on a way to propagate DQ flags to extract_1d, which will factor into the error propagation discussion (JP-1737).

Issue: How to do the propagation of DQ flags from 2D space to 1D space when running extract_1d – which flag values to propagate, how to handle the possibility of the extraction using partial pixels on the edges of an extraction region (and hence how to handle DQ flags from those partial pixels), etc. Extract_1d intentionally excludes any pixels with serious DQ flags (e.g. DO_NOT_USE) from the sum in a given wavelength bin, and hence to a certain extent there aren’t any DQ flags that pertain to the output value (the 1D pixel is “clean”). On the other hand, there may be situations where pixels in the 2D image have one or more non-zero DQ flags set, but they’re treated as non-serious or information only and hence DO_NOT_USE is not set for them and the pixel gets included in the 1D extraction. In those cases it might be desirable to carry over the informational flags so that users know that there was something perhaps not quite right with the inputs.

• How do you define a DQ map if you are using, e.g., 25 pixels to extract the flux?
• Not all informational flags would be very useful (RTN?) – which ones are? CRs, saturation?
• What about resampled data? A large fraction of data coming into extract_1d are resampled, so handling DQ flags is impossible because there are no incoming DQ flags.
• What about cube_build? Do 3D cubes coming out have DQ arrays?

Teams, please think about this and provide feedback in the comments (we can have a follow-up discussion later, if needed). {task:id=1415}NIRISS provide feedback on DQ propagation for extract_1d (Kevin Volk, Jo Taylor){task}{task:id=1416}MIRI provide feedback on DQ propagation for extract_1d (Misty Cracraft, David Law){task}{task:id=1417}NIRSpec provide feedback on DQ propagation for extract_1d (James Muzerolle, Maria Pena-Guerrero){task}{task:id=1418}NIRCam provide feedback on DQ propagation for extract_1d (Alicia Canipe, Bryan Hilbert, Nikolay Nikolov){task} Others? Nestor Espinoza, Vincent Geers, Swara Ravindranath, Anton Koekemoer  

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

Hi Alicia Canipe, quick question on this (which could inform my thoughts on feedback for this): when you write "(...) Extract_1d intentionally excludes any pixels with serious DQ flags (e.g. DO_NOT_USE) from the sum in a given wavelength bin" --- do you mean Extract_1d really omits it, leaving that column with one less pixel added to the entire sum? So if I have a two-pixel aperture for the entire spectrum, but one of the columns has one of those two flagged as DO_NOT_USE, then that column will have the contribution of only one pixel?

As a follow-up comment, this might also inform what to do with flagged pixels by the outlier_detection TSO3 step, for which we believe it would be good to have a separate DQ flag. For TSOs, for instance, instead of just "removing" pixels, my view is that it would be better to re-sample them using frames before and after the frame that has a pixel flagged by the DQ flag. We have planned discussions for this in the TSO WG for the future --- but right now we are focused on testing the outlier_detection algorithm first.

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

As I discussed in the brain-storming session, I think there is utility of propagating some small set of specific flags from the 2-D spectral cut-outs to the extracted spectrum for when a pixel is affected by a cosmic ray or where the ramp saturates.  The utility of this is to warn the user that the slope measurements in these pixels may be less accurate.  The values are not known to be bad, but there is a possibility of some inaccuracy.  This can be done for the resampling as well, propagating these specific flags to the resampled image if the individual pixels that contribute have these flags.  The flags would be strictly for information purposes.  There is no need to propagate all the individual flags from the 2-D image pixels to the output spectral pixels.

Aside from the jump and saturation flags on the ramps, one might want to have a flag for the spectral pixels where some of the input values in the 2-D spectra are bad and have slope of 0.0.  This is more of a concern in those cases where there is no resampling of the 2-D image, or if the number of dither positions is small.  If one has a few dither positions in the 2-D images then the resampling should remove most of the effects of these pixels.  In the example plots posted by Swara it is not clear whether the dips in the individual spectra are due to bad pixels that are not flagged or by bad pixels that are flagged and give signal 0.0 so reducing the extracted signal when the simple boxcar extraction is done in the pipeline. A bad pixel near the peak of the trace would remove a significant amount of signal from the spectrum.  This could be improved by interpolating the signal to the bad pixels, or by any type of profile fitting in the extraction.

 

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Howard Bushouse would you mind chiming in on Nestor's question above? I don't know offhand what the code does exactly for those cases. 

[stscijgbot-jp]

Comment by Howard Bushouse on JIRA:

Nestor Espinoza looking over the low-level extract_1d code and running a test with data manually set to 'DO_NOT_USE' I can confirm that this is exactly what's happening. Pixels flagged as 'DO_NOT_USE' are simply excluded from the sum within a given wavelength bin (image column) and hence the total flux and/or surface_brightness is underestimated, because there's no accounting for the missing data (i.e. no interpolation, no renormalization, etc.). You simply get less flux. The only warning to the user right now is the fact that the value in the 'NPIXELS' column of the x1d table will reveal that fewer pixels were used in the sum. So perhaps this is motivation for putting some kind of DQ flag in the x1d column for any wavelength bins that've had pixels excluded, because the resulting flux is essentially unusable.

[stscijgbot-jp]

Comment by Nestor Espinoza on JIRA:

Hi Howard Bushouse --- thanks for confirming this!

I agree that putting a DQ flag would be useful, and that what the pipeline is doing right now is problematic at least for TSOs. Because some measurements are differential (e.g., for transit/eclipse/phase-curve exoplanet science) if that same x1d column has that pixel flagged in every integration then this might not be a problem if you analyze every lightcurve at the column-level (i.e., if N pixels decrease 99% of their flux during transit, N doesn't really matter once you divide your lightcurve by the total out-of-transit flux). However, most folks will be analyzing wavelength (i.e., x1d column) bins, on which this current approach at extracting the spectra is troublesome (it weights the differential lightcurve outside of that particular wavelength/column bin).

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Discussion in DMSWG about DQ propagation: https://outerspace.stsci.edu/pages/viewpage.action?spaceKey=JWSTPWG&title=2021-05-12+Meeting+notes%3A+DMSWG 

Decision: 

• Baseline: Stick with what we currently have, which is the DO_NOT_USE bit. Other flags are informational and are not used consistently yet.
• Enhanced: We can figure out if there is other information to carry over once we have flight data.

I'll create a ticket for an "enhanced" version of this to start thinking of a better way to track the information than excluding DO_NOT_USE and capturing that pixels were excluded in the NPIXELS column. 

We'll have a follow-up DQ-focused discussion about how INS is using flags and how to clean that up in general. 

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Since teams agreed to stick with the baseline version of only using DO_NOT_USE flag for now, I'll close this ticket and we can track improvements in the JP-2924 ticket. 

[stscijgbot-jp]

Comment by Alicia Canipe on JIRA:

Closing this ticket after discussions about using only DO_NOT_USE for now, and enhancements will be tracked in JP-2924.