Stricter flagging of bad pixel neighbors

[bhilbert4]

Another tweak that I have made while looking at NIRCam bad pixels is to look at neighbors of pixels flagged by RCSelfCal using a stricter threshold.

I've seen the best results so far by running RCSelfCal with a 5 sigma threshold, and then going back and looking at the 4 neighbors around each flagged pixel and flagging those using a 3 sigma threshold. I currently do this neighbor flagging outside of RCSelfCal, but I could look into integrating it if you think it would be useful.

[jdavies-st]

That could be interesting. This is essentially a flagging of the IPC effect? I.e. if the central pixel is large outlier, then it's likely that the connected pixels due to IPC will be some ~0.6% of that or whatever the IPC kernel is?

That makes sense. I've mostly been relying on JumpStep to catch these, but if these are consistently hot pixels that just have a much higher baseline (bias), then maybe they don't get flagged, nor do the pixels around them, is there's actually no jump in the slope. Is that what you've seen as well?

[jdavies-st]

Here's what a median stack of NRCALONG looks like. You can see the little "flowers" or OPEN pixels and ADJ_OPEN, or something close to those, and you can see the hot pixels with an IPC effect at their adjacent ones too.

For most of these "flowers" already have the central pixel flagged as BAD and set to NaN. The ADJ_OPEN ones around though are not.

It's possible that that perhaps since these central pixels are getting caught early, and flagged as bad during the jump step and not used in the ramp fitting, perhaps these are actually also hot pixels that are causing IPC? Would be worth looking at high-zodi final groups in the _uncal files.

[jdavies-st]

To answer my own question, here's a blink between the first group of individual exposure

jw02234001001_02101_00002_nrcalong_jump.fits

and the median stack made from all NRCALONG exposures in program 2234. I've colored the NaNs in the stack green, as they are NaNs in the input rate files (due to how they're handed in RampFitStep). But you can see that these are OPEN and ADJ_OPEN pixels, or something very similar.

[bhilbert4]

Yeah, from what I've seen, most of the flower centers are NaN in the rate files. I've seen some that are true RC pixels with various time constants, with the signal high in the initial reads, and fading as the ramp goes on. In that case, the number of jump flags up the ramp is high enough that the ramp fitter gives up and calls it NaN. In those cases, it makes sense that the neighbors look hot, due to IPC, and so you get flowers.

I've also seen cases where the central NaN pixel has only 1 or 2 neighbors that have an elevated signal rate, while the other neighbors are ok. That was my motivation for looking at neighbors with a stricter sigma limit, rather than blindly flagging all 4 neighbors.

[jdavies-st]

Interesting. It sounds like there's a number of electronic effects going on.

I was actually quite surprised that the first group shows these pixels so well. It turns out so does the ZEROFRAME. Would be interesting to use a stack of the ZEROFRAMES to get map of these bad pixels.

[bhilbert4]

A good number of the RC pixels have a very high signal rate in the early reads. I've also seen some that have a very short time constant, such that the signal between frame 0 and frame 1 can be thousands of counts, but between consecutive later frames the rate has decayed to nominal dark current level.

[jdavies-st]

Interesting. What's the current strategy for handling these? Masking them? Or will an RC model be used to correct them? Or just mask the first N groups until they've settled down?

[bhilbert4]

Mask them. They're few enough that it's not worth trying to save them at the moment. Who knows what other symptoms might show up in addition to/because of the RC behavior? It's safer to just throw them out.

[jdavies-st]

Agree. In that case, the ZEROFRAME images are very good for finding these. Here's a thumbnail cutout of the median of 40 NRCALONG zeroframe images from program 2234.

[bhilbert4]

Coming back around to this now that I've delivered updated mask files to CRDS, and am looking to clean up and improve my process for next time. This time around, I really wanted to get something in for reprocessing associated with the next build of the pipeline, and it seems like what I had was most of the way there.

For the pixels that are dark in the zeroframe/0th group, have you looked at any signals up the ramp? I'd always been thinking that these pixels were dark because they were saturated in all groups, which would then make them NaN in the rate file. But if they are showing low, finite (or exactly zero?) signal values in the initial group and the rest of the way up the ramp, then I guess that's not the case.

I can put in a PR to add the neighbor search, if you think it will be useful. Like was mentioned in a conversation with some other NIRCam folks a few weeks ago, since the IPC step is off by default, most users' data is not IPC corrected, which means very hot/RC pixels can cause their neighbors to also appear hot/RC, just at a lower level.

And then it seems like these dark central pixels would be a separate search, and that OPEN/ADJ_OPEN would be the flag to use if they have low/zero signal up the ramp.