Excess Jump Step Flagging in NIRISS data

[stscijgbot-jp]

Issue JP-3737 was created on JIRA by Rachel Plesha:

During an investigation about odd behavior around the edges where the jump step was flagging the edges in a sort of scalloped pattern with pipeline version 1.14.0 (2883005_001_dq.png), we encountered an apparent bug with the new build 11.0 (1.15.1) version of the pipeline. If the data are calibrated with pipeline version 1.15.1, the scalloped edges remain, and additionally there is severe over-flagging most of the exposure (IMAGING_jw02883005001_03201_00001_nis_rate_difference.png). We had seen some of this behavior in our testing of the build version, but to a much lesser extent (WFSS_jw01510001003_0210i_00001_nis_rate_difference.png).

This gives us two puzzles: 1) what are these scalloped edges, and 2) why is this over-flagging occurring at this scale for this exposure (and potentially others)? Both of these effects appear to be related to the jump step as they are flagged with DQ=4 (JUMP_DET), but we do not understand under what circumstances this happens. Some updates to the snowball flagging were made as part of build 11.0 (JP-3638 & JP-2622), which could be causing the over-flagging.

We do not currently know if this affects users, but this does not appear like it is working as intended, so we are raising the concern now.

[stscijgbot-jp]

Comment by David Law on JIRA:

Huh, well that's weird.  Poking at it a bit, something funny seems to be going on with the science data regardless of the jump step flagging.

diffs.png attached shows group-minus-group differences of the raw data as it's getting sent into the jump step (i.e., after correction for refpix, superbias, dark, etc.  Differences 1, 3, and 10 (indeed, all but one of the the differences) have median values about 10 or fewer DN/s.  Difference 8 though has a median difference of about 30 DN/s.  Something happened that made the entire array brighter by significantly more than the usual amount in one of the groups, and it's this group in which the large number of big circles are getting flagged (presumably by the CR shower routine).  Depending on whether or not the shower routine flagged the data in this group the final slope can be quite different.

I'd say this isn't a bug in jump per se, but it's just telling us about whatever weird effect caused the entire detector to brighten suddenly.  Micrometeorite or something?

[stscijgbot-jp]

Comment by Stephanie La Massa on JIRA:

Thanks for looking into this David Law! We did have a case once where half the detector brightened partway through an exposure. Eddie thought it might be due to a glow from an evaporating micrometeorite.

Maybe we should share this data with Eddie and see what he thinks?

[stscijgbot-jp]

Comment by David Law on JIRA:

Sure, worth bouncing off of Eddie Bergeron to comment on whether this may be a similar case.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Yes, this is indeed a micrometeor impact flash! About 30 DN amplitude in NIRISS, just barely above the 1/f amplitude, and it occurred just after the start of group 9 of this integration. To my surprise you can see it in the ERR and VAR_RNOISE in your figures above as a darker vertical band on the right-hand edge. That dark band was before the flash, and the flash signal is seen left of that. You can see it in CDSs up the ramp of the uncal, but it's really hard to see due to the low amplitude.

I pulled the FG data and it is easily detected there, with a nice, well-sampled tail. Plots attached.

The easiest solution for calibration of this set would be to run the pipeline through jump, set CR or do_not_use for all pix in group 9 of the jump output, then continue with ramp fit. Alternately you could simply subtract 30 DN from the 9th group of the uncal before processing, or a variation on this with a step down for the band on the right side of group 9. I recommend just throwing out the group.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

The manifestation of the flash in the rate image here is interesting. The flash itself was too faint to trigger a frame-wide jump, but the snowball detection turned it up due to the large snowball halo ellipses. The large area of the correctly flagged snowballs makes the sudden slope change partway through the ramp really stand out.

[stscijgbot-jp]

Comment by Rachel Plesha on JIRA:

I wanted to document here that André Martel found what appears to be another example of this in PID 2078 obs 15 (jw02078015001_02201_00006_nis_cal.fits).

From André:

I attach an image of 2078:15 (left) and the 2883 image on the right. The 2883 image seems to have issues mostly on the left edge while 2078 is on the right edge (vertical striping, elongated structures, etc…).

!meteor_2078_2883.png|width=600,height=300!

[stscijgbot-jp]

Comment by David Law on JIRA:

Looks like this is a similar case.

So, the question is what if anything to do about it?

One key point of both may be that these asteroid flashes cause large numbers of pixels to be flagged as JUMP_DET across the entire array.  In 2078, it looks like 20% of the array in the affected groups, in 2883 more like 50%.

One possibility might therefore be to add a final line to the jump step; for any group in which > X % of the array was flagged as JUMP, flag the entire array as JUMP.  X=10% should take care of both of these examples.

However, there's always the question of unintended consequences.  Might we cause more exposures to have problems with such a modification than we fix?  E.g., for giant planet observations that can saturate the field rapidly?  For sake of safety, perhaps only for FULL array data that isn't TSO?

Tagging Michael Regan and Eddie Bergeron to get their thoughts.  Would so simplistic a hack be practical to explore?

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

I had a look at jw02078015001_02201_00006_nis_uncal.fits and the 7 FGS fg datasets for this exposure. It does look like a flash in the niriss dataset, slightly fainter than the 2883 case, however I don't see any signs of a flash in the fg data. Also Marcio's master list of detected flashes in 2022+2023 does not show a flash for this dataset. I'll ask him to take a closer look at this one.

Last week I did ask Marcio to run his algorithm on the 2024 sets and he did find the 2883 flash:

_Hi Eddie, Yes, I see it! I just updated my list of bright flashes. jw02883005001_gs-fg_2024228172209cal bright flash at observatory time 2024-08-15 17:18:16.583

 

Marcio keeps an online archive with all the fg data ever taken since launch and has been refining his code to search for flashes. He is also adding the ACQ and TRACK data. I think we have just over 200 flash detections at this point. That's in ~2.5 years on orbit.

The FGS FG data is optimal for detecting flashes since it is always in use when taking science data and is unfiltered 0-5um. We have seen a few in FG where there is no corresponding science data (i.e. between exps or during reset frames). If we eventually want to flag these for use by the pipeline or anything else I think the fgs fg data is the best way to go. As you can see with just these 2 NIRISS datasets there isn't a sure way to detect them based on snowball flagging. Flash brightness and location within a ramp will matter. Not sure why I'm not seeing anything in the fg data for 2078, but for all* the science cases I've looked at, the flash was also detected in the fg data.

*the one other exception to this might be the very first flash discovered...by the NIRISS team. A very faint one indeed, but I don't think I ever went back to confirm that one in fg. Will do that and get back to you.

[stscijgbot-jp]

Comment by Rachel Plesha on JIRA:

André Martel found another example of this overflagging to add to the sample size. It seems like this is common enough, that it might make sense to do something about it like you suggest, David.

I think I’ve identified a third meteor hit or flash in the NIRISS data. This is an exposure of the CANUCS program: jw01208010001_20201_00001_nis_cal.fits. The left edge seems the most affected (vertical stripes, elongated structures, etc…) and it seems much weaker than the previous examples. I attach a figure of this image (left panel), and for comparison, the other two found earlier:jw02078015001_02201_00006_nis_cal.fits (middle) and jw02883005001_03201_00001_nis_cal.fits (right panel).

!meteor_1208_2078_2883.png|width=800,height=300!

[stscijgbot-jp]

Comment by David Law on JIRA:

Looks like the code above with X=10% takes care of jw01208 as well.  Not the bright vertical streak though, my guess is that's an artifact from a bright out of field star based on the tilt.

Runtime of the code is about 1 sec even for 100-group, 3-int MIRI data, so negligible in most cases.

I'm going to push the code to a PR so that we can test it more broadly: spacetelescope/stcal#308 and #8894

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Yes, this one is already on our list, detected in FGS by Mario's routine. My notes say:

jw01208010001_gs-fg_2023057174614_cal   **  FLASH (BIG)  *****

It's an interesting flash - fairly bright with an unusually long tail. Plots attached.

So we've detected some 200 of these so far between launch and early 2024. If you assume these are spread evenly across 3 SIs and ignore parallels, you should expect to see ~30/yr/SI. Not sure if the flagging issues will be niriss jump threshold specific. This one is bright enough (~30 DN) that it probably should have been detected by jump. Note that the number of flashes as a function of brightness is following a power law - there will be many more faint ones than bright ones and this 200 number is an underestimate. Mario is still working on pushing his detection threshold down.

Looking at the uncal in this case, there is an interesting pattern in the illumination. In addition to the uniform jump there is a bright-ish band along one side. You can see this as a slightly darker band across the top of your 3-panel figure. Not sure what this is. Could be an internal reflection inside niriss or could be related to the long tail of this flash. Maybe it's a ricochet or it hit something other than the primary - maybe secondary or baffle or truss. I'll try to clean it up a bit. Do you know off-hand if there were parallels?

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

I presume there's a different jump threshold for grouped vs ungrouped data? Sensitivity to these will depend both on the flash brightness and which group of the grouped frame the flash occurred in (just like any other "CR").

 

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Just spoke to Mike about this. The threshold doesn't change, but the readnoise is scaled by sqrt(n-grouped-frames). This should be fine.

I'm concerned about using the %jump detections to detect these. It's not really the right way to go about it, since other things like 1/f can juke this around. Perhaps another iteration of jump detection that compares the median of the entire frame in CDSs up the ramp. Since the flash signal covers the whole frame, treat the whole frame as a single pixel and look for jumps that way too?

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

For example, using this dataset and just taking the median of each of the raw frames (in this case after a refsub correction). This plot shows the medians in white and the differences of those medians in red. Figure attached.

[stscijgbot-jp]

Comment by Rachel Plesha on JIRA:

I'll defer to others' expertise about how to go about fixing this, but I will mention that 1208 obs 10 is a coordinated parallel.

I'm also not getting notifications currently for comments for this ticket for some reason (Stacy is looking into this for me), so if I don't respond for a while that's likely why, and feel free to ping me on slack to take a look.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

Great, I'm very interested in the parallels. Will take a look. This flash is a strange duck!! Took me a while to realize that in addition to that bright flash band across tehe top, the flash signal across the whole array is also sloped upwards from bottom to top towards that bright band. In addition, the band itself is slightly diagonal and is increasing in brightness as the flash progresses. If I had to speculate I'd say that a warm piece of impact debris drifted away from the impact site, changing the illumination spatially as it went. If I stick my neck way out, it even looks like that bright band is a trailed moving donut, though the odds of an out-of-focus object on a track almost perfectly aligned with the niriss rows seems remote. Figures attached showing the flash signal CDS and some surface plots.

[stscijgbot-jp]

Comment by David Law on JIRA:

Eddie Bergeron I'm not sure that looking for outliers in the median differences of the raw frames would be robust.  In the example here we'd want some automated way to clip out the bad group, but the outlier difference is sufficiently close to the rest of the differences that using 2sigma , 2.5sigma, or 3sigma clipping to determine the group to reject makes a difference of rejecting 0, 1, or 2 groups.  Likewise, I'd be concerned that the median group difference could change for bright scenes that saturate across a large fraction of the detector somewhere on the ramp.  If we include the saturating pixels in the median, the median will change since we're no longer accumulating signal.  If we omit the saturating pixels from the median then the set of source pixels is changing with group number, and so the median can differ too.

1/f correction happens after jump, so I agree that could be a concern.  I'll try to get some statistics on the 1/f test cases to see what fraction of pixels in a given group is typically flagged there.

[stscijgbot-jp]

Comment by Eddie Bergeron on JIRA:

You could use the final saturation and CR map to make a single mask that is used for the median of each frame to keep the set of source pixels constant. Still have sigma issues to consider though. 

 

I guess the most robust option is still to sync the fgs fg clock to the science data and use the clock time of the flash detection in fgs to decide what frame to reject. But that's rapidly leaving the realm of "simple corrections."

[stscijgbot-jp]

Comment by David Law on JIRA:

For sake of interest I worked up the statistics for all observations in the 1/f test data set, along with some others, calculating the sigma-clipped mean and rms fraction of pixels flagged as JUMP in a given group, along with the MAX number.  fraction.png attached shows the results; all the exposures that I've tested so far are very consistent, peaking at a max of about 0-2% pixels flagged in a given group, with the exception of the three test cases with micrometeorites.

If you can point me to a few other cases identified via FGS I'll add them to this plot and see if it can detect them.

[stscijgbot-jp]

Comment by David Law on JIRA:

Tried this out on jw01208010001_14101_00003_nrca1_uncal.fits, which is a NIRCam DEEP8 observation in which the FGS recorded a flash.  The rate image shows a similar swiss-cheese appearance where the flash is masked out by a pair of snowballs, but in this case the flagged fraction covered only about 5% of the FOV.  Likewise, the median difference across the image didn't change a great deal from group to group (Just 4 group differences, two fairly low and two higher).  Other cameras taking data at the same time didn't show swiss cheesing, maybe just no snowballs in these.

So, clear that neither the masked-fraction nor the average group differences will reliably pick out these events in all cases.  Perhaps that's ok though, as we can live with missing the subtle cases

[stscijgbot-jp]

Comment by Kevin Volk on JIRA:

The snowball halo flagging code does treat the edges of the detector differently than the central area, so this is likely to be part of the cause of this unusual flagging. There is the question of exactly what thresholds to use for the snowball halo flags; some tests were made in a previous version of the pipeline to set the values, but it was difficult to tell what the "best" parameters were.