Remove diaSrc objects from association based on a set of flags

[rearmstr]

It would be useful to potentially filter out some diaSrc objects when running association based on a set of flags. This could help to remove some of the junk sources in the diaSrc catalogs.

[wmwv]

@shuliu2017 What's your recommended set of flags?

[shuliu2017]

@wmwv I am using base_PixelFlags_flag_bad, base_PixelFlags_flag_saturatedCenter, base_PixelFlags_flag_saturated, base_PixelFlags_flag_edge, base_PixelFlags_flag_offimage.

[shuliu2017]

Below are all the PixelFlags I got from the source table, I am not sure whether we should also include base_PixelFlags_flag_cr and base_PixelFlags_flag_crCenter.

base_PixelFlags_flag base_PixelFlags_flag_offimage base_PixelFlags_flag_edge base_PixelFlags_flag_interpolated base_PixelFlags_flag_saturated base_PixelFlags_flag_cr base_PixelFlags_flag_bad base_PixelFlags_flag_suspect base_PixelFlags_flag_interpolatedCenter base_PixelFlags_flag_saturatedCenter base_PixelFlags_flag_crCenter base_PixelFlags_flag_suspectCenter

[wmwv]

Yes, I support adding flag_cr and flat_crCenter to the rejection flags.

[shuliu2017]

Here are all the flags I got from the diaSrc table using the code:

dia_astropy = dia_src.asAstropy()
for col in dia_astropy.columns:
    if re.search('flag', col, re.IGNORECASE):
        print('`'+col+'`')

flags_negative base_NaiveCentroid_flag base_NaiveCentroid_flag_noCounts base_NaiveCentroid_flag_edge base_NaiveCentroid_flag_resetToPeak base_PeakCentroid_flag base_SdssCentroid_flag base_SdssCentroid_flag_edge base_SdssCentroid_flag_noSecondDerivative base_SdssCentroid_flag_almostNoSecondDerivative base_SdssCentroid_flag_notAtMaximum base_SdssCentroid_flag_resetToPeak base_SdssCentroid_flag_badError ip_diffim_NaiveDipoleCentroid_flag base_CircularApertureFlux_flag_badCentroid base_GaussianFlux_flag_badCentroid base_NaiveCentroid_flag_badInitialCentroid base_PeakLikelihoodFlux_flag_badCentroid base_PsfFlux_flag_badCentroid base_SdssCentroid_flag_badInitialCentroid base_SdssShape_flag_badCentroid slot_Centroid_flag ip_diffim_NaiveDipoleCentroid_pos_flag slot_Centroid_pos_flag ip_diffim_NaiveDipoleCentroid_neg_flag slot_Centroid_neg_flag base_SdssShape_flag base_GaussianFlux_flag_badShape slot_Shape_flag base_SdssShape_flag_unweightedBad base_GaussianFlux_flag_badShape_unweightedBad slot_Shape_flag_unweightedBad base_SdssShape_flag_unweighted base_GaussianFlux_flag_badShape_unweighted slot_Shape_flag_unweighted base_SdssShape_flag_shift base_GaussianFlux_flag_badShape_shift slot_Shape_flag_shift base_SdssShape_flag_maxIter base_GaussianFlux_flag_badShape_maxIter slot_Shape_flag_maxIter base_SdssShape_flag_psf base_GaussianFlux_flag_badShape_psf slot_Shape_flag_psf base_CircularApertureFlux_3_0_flag base_CircularApertureFlux_3_0_flag_apertureTruncated base_CircularApertureFlux_3_0_flag_sincCoeffsTruncated base_CircularApertureFlux_4_5_flag base_CircularApertureFlux_4_5_flag_apertureTruncated base_CircularApertureFlux_4_5_flag_sincCoeffsTruncated base_CircularApertureFlux_6_0_flag base_CircularApertureFlux_6_0_flag_apertureTruncated base_CircularApertureFlux_6_0_flag_sincCoeffsTruncated base_CircularApertureFlux_9_0_flag base_CircularApertureFlux_9_0_flag_apertureTruncated base_CircularApertureFlux_9_0_flag_sincCoeffsTruncated base_CircularApertureFlux_12_0_flag slot_ApFlux_flag base_CircularApertureFlux_12_0_flag_apertureTruncated slot_ApFlux_flag_apertureTruncated base_CircularApertureFlux_17_0_flag base_CircularApertureFlux_17_0_flag_apertureTruncated base_CircularApertureFlux_25_0_flag base_CircularApertureFlux_25_0_flag_apertureTruncated base_CircularApertureFlux_35_0_flag base_CircularApertureFlux_35_0_flag_apertureTruncated base_CircularApertureFlux_50_0_flag base_CircularApertureFlux_50_0_flag_apertureTruncated base_CircularApertureFlux_70_0_flag base_CircularApertureFlux_70_0_flag_apertureTruncated base_GaussianFlux_flag base_PeakLikelihoodFlux_flag base_PixelFlags_flag base_PixelFlags_flag_offimage base_PixelFlags_flag_edge base_PixelFlags_flag_interpolated base_PixelFlags_flag_saturated base_PixelFlags_flag_cr base_PixelFlags_flag_bad base_PixelFlags_flag_suspect base_PixelFlags_flag_interpolatedCenter base_PixelFlags_flag_saturatedCenter base_PixelFlags_flag_crCenter base_PixelFlags_flag_suspectCenter base_PsfFlux_flag slot_PsfFlux_flag base_PsfFlux_flag_noGoodPixels slot_PsfFlux_flag_noGoodPixels base_PsfFlux_flag_edge slot_PsfFlux_flag_edge ip_diffim_NaiveDipoleFlux_flag ip_diffim_NaiveDipoleFlux_pos_flag ip_diffim_NaiveDipoleFlux_neg_flag ip_diffim_PsfDipoleFlux_flag ip_diffim_PsfDipoleFlux_pos_flag ip_diffim_PsfDipoleFlux_neg_flag ip_diffim_ClassificationDipole_flag ip_diffim_DipoleFit_flag_classification ip_diffim_DipoleFit_flag_classificationAttempted ip_diffim_DipoleFit_flag ip_diffim_DipoleFit_flag_edge ip_diffim_forced_PsfFlux_flag ip_diffim_forced_PsfFlux_flag_noGoodPixels ip_diffim_forced_PsfFlux_flag_edge

[wmwv]

I think we should think about flag_edge a bit more. To the degree that it refers to having an edge of a CCD go into the coadd template, but where we have 20 images making up that template, we may not clearly want to throw out such flag_edge objects.

[rearmstr]

I'm wondering if flag_cr is needed. Since we interpolate over cosmic rays, I can see a case where having a cosmic ray in the outer regions of the footprint may have little effect on the measurement.

Another question I have is do you care about tracking what gets rejected by the flag cut? Will there be a case where someone can't find a particular object in an association because it was rejected by a flag. We will have no knowledge of why it was missing unless we know which flags were used by the algorithm which should in principle be available, but can also be hard to find.

[rearmstr]

@wmwv isn't the problem with flag_edge that the object will have missing pixels?

[wmwv]

@wmwv isn't the problem with flag_edge that the object will have missing pixels?

No. flag_edge can be set because the pixels in the template were on the edge of a contributing CCD. But there could be 8 other CCDs that did contribute pixels.

flag_edge as propagated from the calexp indeed represents sources that should be rejected.

[wmwv]

Another question I have is do you care about tracking what gets rejected by the flag cut?

Yes, in the long term, but I don't have a clear vision for how to do this efficiently.

Will there be a case where someone can't find a particular object in an association because it was rejected by a flag.

Yes. But many of those cases it may be in fact correct to reject the object. If, e.g., the center pixel is saturated, we should not count it and that's a legitimate loss of efficiency.

We will have no knowledge of why it was missing unless we know which flags were used by the algorithm which should in principle be available, but can also be hard to find.

But trying to ask (1) "what happened to each fake I put down" is a different question than (2) "what did I recover from the simulations". Question (1) inherently requires a different form of analysis than (2).

[rearmstr]

No. flag_edge can be set because the pixels in the template were on the edge of a contributing CCD. But there could be 8 other CCDs that did contribute pixels.

I'm confused. flag_edge refers to being on the edge of the image. Are you talking about sensor_edge? This indicates pixels on the coadd that had an edge from one of the input images.

[rearmstr]

Yes. But many of those cases it may be in fact correct to reject the object. If, e.g., the center pixel is saturated, we should not count it and that's a legitimate loss of efficiency.

Yes, I'm sure most of these we want to ignore, I was thinking more about the edge cases.

But trying to ask (1) "what happened to each fake I put down" is a different question than (2) "what did I recover from the simulations". Question (1) inherently requires a different form of analysis than (2).

I agree. I'm happy not to worry about it for now. I've just seen many problems that took a long time to track down that were a result of similar choices.

[wmwv]

I'm confused. flag_edge refers to being on the edge of the image. Are you talking about sensor_edge? This indicates pixels on the coadd that had an edge from one of the input images.

Yes, I meant what you describe as sensor_edge. I must have had the flag names wrong in my head. Thank you.