Excess Jump Step Flagging in NIRISS data

[stscijgbot-jp]

Issue JP-3737 was created on JIRA by Rachel Plesha:

During an investigation about odd behavior around the edges where the jump step was flagging the edges in a sort of scalloped pattern with pipeline version 1.14.0 (2883005_001_dq.png), we encountered an apparent bug with the new build 11.0 (1.15.1) version of the pipeline. If the data are calibrated with pipeline version 1.15.1, the scalloped edges remain, and additionally there is severe over-flagging most of the exposure (IMAGING_jw02883005001_03201_00001_nis_rate_difference.png). We had seen some of this behavior in our testing of the build version, but to a much lesser extent (WFSS_jw01510001003_0210i_00001_nis_rate_difference.png).

This gives us two puzzles: 1) what are these scalloped edges, and 2) why is this over-flagging occurring at this scale for this exposure (and potentially others)? Both of these effects appear to be related to the jump step as they are flagged with DQ=4 (JUMP_DET), but we do not understand under what circumstances this happens. Some updates to the snowball flagging were made as part of build 11.0 (JP-3638 & JP-2622), which could be causing the over-flagging.

We do not currently know if this affects users, but this does not appear like it is working as intended, so we are raising the concern now.

[stscijgbot-jp]

Comment by David Law on JIRA:

Huh, well that's weird.  Poking at it a bit, something funny seems to be going on with the science data regardless of the jump step flagging.

diffs.png attached shows group-minus-group differences of the raw data as it's getting sent into the jump step (i.e., after correction for refpix, superbias, dark, etc.  Differences 1, 3, and 10 (indeed, all but one of the the differences) have median values about 10 or fewer DN/s.  Difference 8 though has a median difference of about 30 DN/s.  Something happened that made the entire array brighter by significantly more than the usual amount in one of the groups, and it's this group in which the large number of big circles are getting flagged (presumably by the CR shower routine).  Depending on whether or not the shower routine flagged the data in this group the final slope can be quite different.

I'd say this isn't a bug in jump per se, but it's just telling us about whatever weird effect caused the entire detector to brighten suddenly.  Micrometeorite or something?

[stscijgbot-jp]

Comment by Stephanie La Massa on JIRA:

Thanks for looking into this David Law! We did have a case once where half the detector brightened partway through an exposure. Eddie thought it might be due to a glow from an evaporating micrometeorite.

Maybe we should share this data with Eddie and see what he thinks?

[stscijgbot-jp]

Comment by David Law on JIRA:

Sure, worth bouncing off of Eddie Bergeron to comment on whether this may be a similar case.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Yes, this is indeed a micrometeor impact flash! About 30 DN amplitude in NIRISS, just barely above the 1/f amplitude, and it occurred just after the start of group 9 of this integration. To my surprise you can see it in the ERR and VAR_RNOISE in your figures above as a darker vertical band on the right-hand edge. That dark band was before the flash, and the flash signal is seen left of that. You can see it in CDSs up the ramp of the uncal, but it's really hard to see due to the low amplitude.

I pulled the FG data and it is easily detected there, with a nice, well-sampled tail. Plots attached.

The easiest solution for calibration of this set would be to run the pipeline through jump, set CR or do_not_use for all pix in group 9 of the jump output, then continue with ramp fit. Alternately you could simply subtract 30 DN from the 9th group of the uncal before processing, or a variation on this with a step down for the band on the right side of group 9. I recommend just throwing out the group.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

The manifestation of the flash in the rate image here is interesting. The flash itself was too faint to trigger a frame-wide jump, but the snowball detection turned it up due to the large snowball halo ellipses. The large area of the correctly flagged snowballs makes the sudden slope change partway through the ramp really stand out.