DRP Bad Pixel Recheck

[kjl0025]

Is your feature request related to a problem? Please describe. Just wanted to document the email discussion from a while back concerning what was on the old FDD so that it isn't forgotten: "Run additional recheck post-correction to see if any of the bad pixels from cosmic-ray flagging are no longer bad–e.g. warm pixels that were well-subtracted." This is currently not implemented in the DRP and is supposed to be removed from the FDD (if it hasn't been already).

Describe the solution you'd like Email discussion:

Kevin: "Interesting thought. The cosmic masking is designed to statistically exclude these one-off warm or hot pixels because it applies a median filter, meaning in a given row, a cosmic is only flagged if the median of n pixels in a row meet the chosen cosmic threshold value, and the case of n>1 excludes one-off warm pixels that may have been below the threshold used to calibrate the bad-pixel map, which is supposed to catch hot pixels. However, with the TVAC data that was taken for obtaining noise maps for CIC, FPN, and dark current, I had to choose n=1 to remove all traces of cosmic rays on the noise maps, regardless of how low my chosen cosmic threshold value was. (And perhaps there are warm pixels in a region that may beat a n>1 filter length anyways?)

My gut tells me this scenario where a warm pixel not caught by the bad-pixel map that pushes the corresponding pixel in a non-dark frame over the cosmic masking threshold is unlikely, but it is theoretically possible. Plus I haven't actually examined actual data post-dark-subtraction for residual cosmic rays to know how "likely" this scenario is, and I'd rather be over-rigorous than under-rigorous.

From cosmic masking to dark subtraction, we have these steps: -cosmic mask made -nonlinearity correction -convert from DN to e- -divide by EM gain -subtract master dark (which has already been EM gain-divided and is in e-, and the frames used to construct it were cosmic-corrected)

If we did the post-dark-subtraction check, the nonlinearity correction would change the effective cosmic threshold value per pixel relative to what was used originally in the cosmic masking function, so I would rather see this check done in the cosmic masking function itself. We could do this check in the following way:

• Take the master dark (in e- by default) corresponding to the given exposure time and EM gain for the frame at hand, and convert it to DN and multiply by EM gain.
• Then apply the inverse of nonlinearity correction to that master dark since the cosmic masking function happens before nonlinearity correction (and it should). The nonlinearity correction step function does an interpolation on the nonlinearity table, so an interpolation can be done to find the inverse map.
• Then subtract that modified master dark from the frame at hand and perform the cosmic masking algorithm on the resulting frame."

Vanessa: "My take aways for R3.0.1 are (and please let me know if you all disagree) …

Implementing a second BP check as written in the TDD FDD is not advised; we need to update the FDD. It’s possible that by reordering L1/l2a/l2b calibration steps, we might produce a cleaner l2b image. However, we don’t have evidence that an improvement is necessary now, and we don’t know how much the improvement might be. My proposed course of action is to file this away in the “ideas to research later” category (for R3.1 or later), and simply update the TDD FDD to remove the 2nd round of BP checks. I plan to prepare the FDD change request after I return from vacation.

My other take away is that we may want to use different default settings for cosmic ray removal for noise calibration data than for regular observation data. But the underlying CR algorithm could stay the same" ...

...since the keyword arguments in the CR algorithm can be tuned as needed.

Additional context Low priority; just documenting this

[ecady-jpl]

For what it's worth, @nasavbailey's action was completed and this was removed from the TDD FDD in ECR 224639.

[nasavbailey]

Thanks for documenting that, @ecady-jpl . I suggest labeling this as a "won't fix" and closing the issue. It could be re-opened if we find evidence in the future that it's needed.

[maxwellmb]

marking as "won't fix" and closing the issue