Flag and mask handling not (yet) consistent

[scizen9]

Definitions:

• Flag - an 8-bit image specifying the character of a given pixel (see below).
• Mask - a boolean image specifying which pixels should be masked (not used) for a given operation.

Flag values are defined as follows: 0 - normal data 1 - saturated pixel (unaltered) 2 - bad column, hot pixel (fixed) 4 - cosmic ray pixel (fixed) 8 - negative flat value (unaltered) 16 - slice edge (unaltered) 32 - low signal flat (unaltered) 64 - outside slice illuminated area (unaltered) 128 - DAR padded pixel (unaltered)

Currently the python version of the pipeline only implements a subset of these flags and it attempts to keep the flag and mask segments of the astropy CCDData object consistent. This is not actually achieved for the flags that are currently implemented. Ultimately, we don't want this behaviour anyway. The mask should be boolean (True or False, where True means apply the mask) and should be generated based on best data reduction practices from the flag image just prior to the mask being used in a primitive operation.

The issue is that the CCDData mask segment causes data to be masked if it has any value other than 0 and the current way the mask image is implemented causes any and all flagged pixels to be excluded from calculations where they might properly be included.

What we need to do:

1. implement all flag values listed above in the pipeline
2. determine which operations (in the primitives) can use the mask image
3. generate a mask image using a bit pattern to generate 1s (True) where we want values masked
4. provide a default value for the bit pattern, but make it user configurable
5. ensure that both the mask and the flag images are written to disk so the user can verify them and use them for post-processing

The bit pattern used for generating a masked image from the flags is determined by science considerations. For now we can provide a default bit mask and implement a way for the user to use their own bit pattern based on their own science goals.

I recommend a default bit pattern that does not mask the fixed pixels (bad column, cosmic ray), but does mask the unaltered pixels from the table above.

[prusinski]

I just wanted to see if there have been any updates on this issue. From what I can tell, the mask frame has not yet been modified to a purely boolean array in the current code. That being said, when there's a chance, this would be a great feature to have for post-reduction stacking.

[anthonyjlau]

Just to add to this conversation, in FlagSaturation.py in lines 36 to 45, it looks like the flag value for saturated values is incorrectly set (assuming that the flag values above are correct). It seems to be adding 8 to the flag where it should be adding 1.

        # Create flags for saturated pixels
        flags = np.zeros(self.action.args.ccddata.data.shape, dtype=np.uint8)

        sat = np.where(self.action.args.ccddata.data > 60000)
        flags[sat] += 8
        number_of_sat_pixels = np.sum(sat)

        sat = np.where(self.action.args.ccddata.data == 0)
        flags[sat] += 8
        number_of_sat_pixels += np.sum(sat)