LRS TA verification image NaNed in pipeline output

[stscijgbot-jp]

Issue JP-3786 was created on JIRA by Greg Sloan:

The latest pipeline release has NaNed all non-imaging fields in TA verification images. That means that the LRS slit field is NaNed, thus defeating the purpose of the TA verification image.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

This looks like a result of JP-3570; the flat field applied to these data (https://jwst-crds.stsci.edu/browse/jwst_miri_flat_0787.fits) have DO_NOT_USE DQ flags applied to the slit field. The pipeline then ensures that all science pixel values with the DO_NOT_USE flag are NaN. If that flat is applied only to TACONFIRM images (GENERIC subarray?), perhaps a simple ref file update would fix the issue.

[stscijgbot-jp]

Comment by David Law on JIRA:

A few clarifications:

• The slit region looks fine in the rate files, providing a workaround for the time being.
• The slit region of the TA confirmation images has always been masked out in the resampled s2d files (used to be zero in Build 10.2, NaN now)
• The slit region was finite-valued in the cal.fits file for Build 11.0, and is NaN in Build 11.1

The pixels in question have NO_FLAT_FIELD, NON_SCIENCE, and DO_NOT_USE set.  The actual flatfield SCI values themselves are largely bogus.  As such, this is exactly what the changes in JP-3570 were designed to address (having a mix of flatfielded and not-actually-flatfielded but multiplied by semi-arbitrary amounts in the CAL files).

So, what to do about it?

• Flatfields are shared across imaging modes.  CRDS selects on FILTER, READPATT, and SUBARRAY, but not by EXP_TYPE (which might otherwise isolate EXP_TYPE=MIR_TACONFIRM).  We'd want to be careful changing the flags in this region lest we start adding them to imaging mosaics.
• We could just add another CRDS selection criterion, but then there would be more flats to manage that differed only in the under-mask region
• Another complication is that MIRI flatfields aren't all consistent; some flag this region with NON_SCIENCE, while others do not.

I think this needs to go to the MIRI team to converge on a consistent way of identifying such pixels via the DQ array, and then continue the conversation accordingly.

[stscijgbot-jp]

Comment by David Law on JIRA:

Questions for MIRI:

• Is it necessary to provide finite values in the LRS slit field in cal.fits files, or is providing these in the rate.fits files sufficient?
• If finite values are provided for the ~ 40x3 pixel region beneath the MIRI slit itself, do they also need to be provided under the metering structure in cal files as well?
• What flatfield values are appropriate to use in these regions?
• Is it desirable to have the slit region finite-valued in ALL MIRI imaging cal.fits files or just LRS TA confirmation images?  What about resampled imaging mosaics?
• Presumably it would be undesirable to maintain an extra set of flatfields for each filter, just for the case of LRS TA and TACONF exposure types?

The challenge here is that the flatfield step is explicitly zeroing (or more accurately, NaNing) out any regions that are not valid science regions with good flatfields when producing the flatfielded image.  This includes all flags DO_NOT_USE, NON_SCIENCE, and NO_FLAT_FIELD.  That could be reenabled through an update of the DQ arrays in the flatfield reference files, but the slit region would then automatically propagate to imaging mosaics too.

Thoughts Greg Sloan Andreea Petric Sarah Kendrew ?

[stscijgbot-jp]

Comment by Andreea Petric on JIRA:

Many thanks, David, for the clarifications. I am confirming I saw this and will discuss it with Sarah and Greg. We do need finite, correct, and consistently flagged values in the LRS slit field in cal. fits in addition to having them in the rate.fits because some users try custom extraction and background determinations using those values. 

 

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

Thinking about the use cases for these images, I think the main value of the cal.fits files is that they have the WCS assignment (rate images do not), so they are very valuable for checking the WCS of the exposure and the observed coordinates of the target. The ability to do photometry is interesting too, but the calibration that's currently applied to them (i.e. the imager photom file) is currently limited in accuracy due to the slit mask filter which is not accounted for in our imager calibration. 

I think the options are:

• skip the flat fielding step for MIR_TACONFIRM images. presumably that's easy-ish to implement, but also affects the data quality over the whole field
• is there a DQ flag combination that would not NaN out the slit pixels, but also not use them in imaging mosaics? 

Answering a few of David Law 's questions:

• yes, I think there is value in having the slit region finite-valued in the cal.fits files as well as the rate files (as above)
• I don't think we need the whole area under the metering structure, but there could be use cases I'm not thinking of...?
• would def prefer not to have to maintain a full extra set of flat fields. 

 

[stscijgbot-jp]

Comment by Marshall Perrin on JIRA:

Also relevant are various discussions about (potentially, in the future) using the TA Confirm images to help improve aspects of LRS pipeline processing, for instance JP-3307 "MIRI LRS - use TA verification image to determine pathloss correction". I recall also some discussion about potentially using the TA verification to tweak the wavelength solution if the source is not vertically centered in the slit, though I can't find an existing ticket for that.  If any of those enhancements are to eventually happen, presumably it would help to have finite-valued slit pixel data in cal files to help enable that.

Sarah asked me about my uses of these data (as a science user of this mode). I confirm that primarily I was using the data to check the position of the source in the slit, relative to the WCS. (In order to use the WCS and the measured residual offset to more accurately know the position of the science target in the dispersed LRS spectrum images) For now I can adjust my code for the workaround of using the pixels from the rate files plus the WCS from the cal files.  As a secondary use I also extracted photometry from the cal images, but that's a secondary case and I acknowledge is going to have inevitably higher errors due to the slit mask filter, as Sarah noted. 

I see & agree with all the practical concerns raised re: not wanting these pixels included in regular mosaics, nor having to maintain a separate set of flats and adjust the CRDS logic to filter on EXP_TYPE. It's an awkward edge case. 

 

[stscijgbot-jp]

Comment by David Law on JIRA:

It sounds like the solution is this:

1) MIRI delivers updated flatfield files in which the LRS slit region (very roughly 297:305, 306:348) is flagged with NON_SCIENCE but does not have the DO_NOT_USE or NO_FLAT_FIELD bits sets.  Likewise, the flatfield SCI values should have reasonable estimates of what the flatfield is for these pixels so that the data are useful.

2) Cal pipeline would have to make a one-line change in flat_field.py that currently sets anything flagged NON_SCIENCE to DO_NOT_USE.  Will have to study whether this will cause problems for any other instruments/modes.  Given the current architecture, this means checking NIRCam and NIRISS, since the relevant lines don't pertain to NIRSpec.

With those two changes it looks like this should behave as desired, producing values in the cal.fits files but not in any resampled data (since the resample step ignores both DO_NOT_USE and NON_SCIENCE already).

[stscijgbot-jp]

Comment by Sarah Kendrew on JIRA:

Thanks David Law! Let's give Greg Sloan  the chance to digest this issue & comment as well, as LRS lead, before finalising any decisions. 

[stscijgbot-jp]

Comment by David Law on JIRA:

Sarah Kendrew Certainly.

Adding a quick note that it looks like this will not impact any other instruments.  From crawling all 637 MIRI+NIRCAM+NIRISS flatfield reference files used by the latest CRDS rmaps, it looks like none of them currently contain any pixels for which NON_SCIENCE is set but DO_NOT_USE or NO_FLAT_FIELD are not.