Saturation Step does not correctly flag grouped data

[stscijgbot-jp]

Issue JP-3318 was created on JIRA by Michael Regan:

Currently, the saturation step does not work correctly for grouped data. The step compares the group DN level to the saturation level from the reference file. If it is above the saturation level, it flags the group and all subsequent groups as saturated. This does not work for grouped data.

Because the DN value in a group is the average of the frames in the group, when saturation occurs within the group the average value can be below the saturation threshold. This causes the the group to be treated as normal instead of being flagged as saturated.

The solution requires estimating the rate to be able to predict if the last "good" group before saturation should actually be flagged as saturated.

 

 

[stscijgbot-jp]

Comment by Bryan Hilbert on JIRA:

Attached to the Optimal Saturation Flagging page are some slides from a 2018 presentation to the CALWG about a grouped saturation flagging algorithm.

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

Bryan Hilbert I remember the presentation. I wanted to bump up the visibility of the problem because some of the effects are hidden. The missed snowball detections are relatively straightforward to see but including a partially saturated group in a slope fit is quite subtle.

[jdavies-st]

A conservative approach for grouped data might be to flag the group previous to the saturated group as also saturated. This might produce some false positives, especially if GROUPGAP is >1 or is similar to NFRAMES. But perhaps this would be a better baseline behavior than what is done currently?

For the same reason one would want to propagate the JUMP_DET flags to the N+1 group if NFRAMES >1, and possibly dependent on GROUPGAP as well. Again, there will be false positives flagged, but perhaps this is better as baseline?

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

I'd like us to review the 2018 algorithm before we proceed. I'm concerned about the performance.

 

[stscijgbot-jp]

Comment by Michael Regan on JIRA:

James,

  I like the idea of flagging the previous group. But still will miss the group that saturates within the last group.

 

[stscijgbot-jp]

Comment by David Law on JIRA:

Adding a note that some of the most severely affected data from this will be addressed by https://jira.stsci.edu/browse/JP-3593, although the approach currently proposed there does not affect the general case.