Dark artifacts in NIRCam cal images from MAST

[stscijgbot-jp]

Issue JP-3661 was created on JIRA by Ben Sunnquist:

Christopher Clark noticed that his data from PID-3429 observations 4 and 6 have dark artifacts in the same detector positions of bright objects from the previous observation 1 data. However, running with jwst pipeline v1.14.0 fixes this issue and the dark artifacts no longer exist. For example, see attachment, where the right image is from MAST and has the dark artifacts and the left image is the same file processed through 1.14.0 and no longer has the dark artifacts. This example file is jw03429004001_08201_00001_nrcblong_cal.fits.

The data was taken on June 11 2024, around the same time as the Build 10.2 transition, so I was wondering if some strange interaction was happening during processing between obs 1 and 4/6 (e.g. obs 4/6 was flat-fielded with some combo of obs 1 data?).

[stscijgbot-jp]

Comment by Ben Sunnquist on JIRA:

Added another attachment showing the prior observation 1 image that contains the bright sources that show up as dark spots in the following images. The problem only shows up in the cal files, not the rate files.

[stscijgbot-jp]

Comment by Bryan Hilbert on JIRA:

Could this be a problem with the association generator setting obs 1 as a background observation even though it shouldn't be? NIRCam was in parallel with MIRI as prime during this observation (obs 4). Obs 1 is a MIRI background observation.

[stscijgbot-jp]

Comment by Ben Sunnquist on JIRA:

Yeah I think that could be the issue - I see the MAST version ran the background step in image2 whereas my v.1.14.0 reduction didn't.

[stscijgbot-jp]

Comment by Bryan Hilbert on JIRA:

The target used in Obs 1 is designated as a background target. So in that respect it makes sense that obs 1 would be treated that way for both MIRI and NIRCam.

[stscijgbot-jp]

Comment by David Law on JIRA:

Ben Sunnquist Bryan Hilbert Do you think this requires any particular action to be taken?

[stscijgbot-jp]

Comment by Bryan Hilbert on JIRA:

It might be worth updating association generator rules such that a background observation taken with a parallel instrument is not added to the association file as a background? Or more directly, that a specified background observation is only considered a background for the prime instrument? The parallel instrument won't be pointing anywhere near the specified background target coordinates, and so there's a good chance it won't work as a background observation anyway? Not to mention that I don't think I've ever seen a NIRCam program that includes a background observation. Mario Gennaro Martha Boyer, what do you think?

[stscijgbot-jp]

Comment by Mario Gennaro on JIRA:

I am not sure what the best course of action is here.

The APT documentaion tells us which templates REQUIRE a background linked observation: https://jwst-docs.stsci.edu/jppom/targets/fixed-targets#FixedTargets-ReqBackground&gsc.tab=0  (bottom of the page).

The culprit observation here is MIRI imaging (prime) + NIRCam Imaging (coordinated parallel). Neither one of those templates require a background observation, but such an obs can be successfully specified (and APT will give the following warning, which it does in this case):

"{}Warning{} (Form) Target requiring background exposure selected for template that doesn't require background exposure The target used in this observation was specified as requiring a background exposure. The current observation template does not require the use of a background exposure, as it is not an expected use case for background targets. There is no issue with using the target for this observation without a background exposure. If background exposure is desired, add observation of background target with similar exposure in non-interruptible sequence."

 

Now the user really wanted a background observation for MIRI, but indeed the background subtraction was applied to both NIRCam and MIRI.

I am not sure that there is a simple solution that would make ALL users happy, here are some options:

• do nothing (my favorite)
• maybe there is a way to make it more prominent whether the background subtraction was performed or not? (maybe in the file name?...that would be opening a pandora's box, though..).
• align the pipeline behavior to the APT behavior, i.e. perform the background subtraction only for templates that do require it (in this case the user would have had to reprocess the MIRI obs, and not the NIRCam ones). This would however need either a change in the way APT exports the asn candidates or some override in the pipeline to ignore the background link for "non-REQUIRING" templates

[stscijgbot-jp]

Comment by David Law on JIRA:

Mario Gennaro I'm inclined to the 'do nothing' option, as the pipeline is operating as specified by the user.  However, it is actually the case that NIRCam should never do background subtraction by default?  If so, then a trivial solution may be to update the pars-image2 parameter reference file to set background_subtraction.skip=True

[stscijgbot-jp]

Comment by Mario Gennaro on JIRA:

Yes, NIRCam NEVER recommends the use of a background linked observation (i.e. there is no single NIRCam mode requiring it in APT, although is is always optional, with a warning), so I think that a default of background_subtraction.skip=True would be an acceptable solution.

Just to confirm, this would be the default for MAST products, right? Users can always override this for unique, non-default, non-standard local use, right?

Bryan Hilbert Ben Sunnquist Alicia Canipe , would you agree ? We would need to document this, obviously, and it would affect almost NO observation (except this program and possibly others that may have used NIRCam for a parallel, while doing other instruments in prime for which a background obs was selected).

 

[stscijgbot-jp]

Comment by David Law on JIRA:

Mario Gennaro Yep, users could always change that, this would just be the default.

[stscijgbot-jp]

Comment by Tyler Pauly on JIRA:

-Question for the INS folks here - is it ever expected that a program will want background subtraction using data products from another instrument? That seems unreasonable to me, but I don't know if I'm missing something. I think Bryan's suggestion to not build associations with a background member from a different instrument is also valid, if perhaps a bit more work.-

Thanks for clarifying, Mario!

[stscijgbot-jp]

Comment by Mario Gennaro on JIRA:

Hi Tyler, the issue here is that we had a "background link construct" in APT for a set of 2 coordinated parallel observations. The linkage was requested for MIRI, but was automatically applied also to NIRCam (because NIRCam and MIRI were the 2 coordinated parallel instruments and the links are at the observation level, which includes both instruments).

There was no mix&match of instruments. As you said, we would never want that, but to be clear, that didn;t happen. A NIRCam background observation was subtracted out from a NIRCam "science" observation, but due to the nature of the 2 observations, we ended up with undesired artifacts. These are due to the fact that the background pointing was selected to favor MIRI background subtraction, while the NIRCam coordinated parallel background pointing had a big galaxy in it, that resulted into artifacts.

[stscijgbot-jp]

Comment by Ben Sunnquist on JIRA:

Mario Gennaro Yeah I think skipping it by default is fine. This situation has occurred in only 6 other observations so far but it has always had issues when the background step runs on them, so skipping it seems like a better default. Here's the other examples {program}_{obs}: 1288_14, 1288_17, - "background observation for" is empty in APT so no bkgsub occurred 2128_2, - bkg for obs 1; bkg oversubtraction present 2180_11, - bkg for obs 10; bkg oversubtraction present 3429_1, - bkg for obs 2-7; bkg oversubtraction present 3435_3 - bkg for obs 2; obs 2 doesnt exist yet

[stscijgbot-jp]

Comment by David Law on JIRA:

We'd considered a solution above of using the pars-image2 parameter reference file for NIRCam imaging to set background_subtraction.skip=True

This will require an updated parameter reference file delivery, so I'm assigning this back to the NIRCam team.