HAP detection image should exclude CR-contaminated filters

[stscijgbot-hstdp]

Issue HLA-1138 was created on JIRA by Rick White:

The detection (total) images used to create HAP catalogs are supposed to exclude single-exposure filters whenever possible.  Including those filters leads to CR contamination in the catalogs.

The logic (as implemented in the HLA processing) is that filters with only a single exposure are omitted from the total image unless all filters have single exposures.  In that case, all filters are included.

I thought this was implemented in the pipeline (and maybe it was in the past?)  But I'm looking a recently processed catalogs and see cases where the total image incorrectly includes the N=1 filters.  As a result the catalog is severely contaminated with CRs.  Details will be given in the comments.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

The attached screenshots show the images for visit hst_15194_02_wfc3_uvis_idnd02. This visit has 3 filters: f814w, f656n and f606w. The f656n filter has only a single exposure and is contaminated with CRs. The other two filters have multiple exposures so CRs have been rejected.

The 3 images below show the problem.  The left panel is a color image using only the f814w and f606w filters. It is good quality, with no CR contamination. The center panel is a color image using all 3 filters; the green CRs that come from the f656n filter are apparent. The right panel shows the total image. It clearly includes the f656n filter image, and as a result it is heavily contaminated by CRs.

!hst_15194_02_wfc3_uvis_f814w_f606w_idnd02.png|width=33%! !hst_15194_02_wfc3_uvis_f814w_f656n_f606w_idnd02.png|width=33%! !hst_15194_02_wfc3_uvis_total_idnd02.png|width=33%!

The image below shows the catalog for this image, zoomed in to show a smaller portion of the image. Most of the sources are located on the green cosmic rays from the f656n image. This catalog is so heavily contaminated with CRs that it is not useful.

!hst_15194_02_wfc3_uvis_idnd02_segment.png|width=50%!

As the description says, the total image should omit the single-exposure f656n filter. That will lead to a good quality catalog.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Here is another example of the problem in the current HAP catalogs. Visit hst_9857_01_acs_wfc_j8rh01 has 3 filters. Two of them (f660n and f658n) have multiple exposures to remove cosmic rays. One (f550m) has only a single exposure and so is filled with CRs.

The total image should exclude the f550m image, but it does not. There are lots of CRs in it. The result is a catalog that is a mess. Here is the image plus catalog for the f660n filter using the HLA interactive display. Here is the image plus catalog for the f550m filter. The f660n catalog has 1,545 unflagged sources (that's after fixing some of the other bugs in the flags). The f550m catalog has 11,271 sources. Practically all of the 9,726 additional sources are cosmic rays in the f550m filter.

The extra junk in the catalog would disappear if the f550m image were excluded from the total image.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Rick White

I have addressed this issue, but now I am debating with myself if I have fixed it in the best way.  At this time, the single filter image is ONLY excluded from being drizzled as part of the total detection image as this ticket suggests.  The resultant total images look very fine.  However, the single filter image is included in all other aspects of the SVM processing which produces some inconsistency in the output.  I will note that I might be over-thinking this issue. 

What I am saying is by ONLY excluding the single filter image from the generation of the total image, all of the data products still reflect information in their headers as if the single filter image were part of the total image with regard to footprint WCS and any statistical computations (except the total image which does not list the excluded image as a constituent).  I just want to make sure this is what is wanted.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Michele De La Pena Hmm, I'm not sure. You say that the footprint WCS looks the same as if the single filter were included. I assume that means that the region for the drizzled images is the same as before. So it includes space for all the filter images, even if some single-exposure filters are excluded from the total image. That would be correct. So it can happen e.g. that the total image does not fill up the region allowed for the whole collection of filters.

On the other hand, there is a S_REGION keyword in the header of the total image (and the other images). I assume that will in fact describe the area covered by data in the total image, excluding any area that is only covered by the single-exposure filters. If the total S_REGION keyword were to include the regions covered by those unused filters, that would be a mistake.

Are there other statistical calculations that you are wondering about, where they include all filters but maybe they should apply only to the exposures actually used for the total image? E.g., if there is a total exposure time in the header of the total image, that ought to be the exposure time for just the filters that got used.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Rick White 

Sorry, I was not clear.  They way I have resolved the problem may be incorrect.  The footprint is the same because it is computed based on all of the images in a previous step in the process. I can recompute the WCS just for the total detection image. 

From your response, I see that I need to make more adjustments so the information (i.e., FITS header keywords) in the total detection image reflects what is truly in the total detection image.

I just want to make sure that you do not mean I should exclude the single filter image entirely from the SVM processing.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

Michele De La Pena

The single-exposure filters should be excluded only from the detection image, but should be processed the same way as before for everything other step. So specifically:

• Compute the drizzle image coordinate limits (WCS I guess) using all the filter images.
• Drizzle all the filter images and exposures to make the drc/drz products.
• Create a detection (total) image that uses the same WCS as all the other images, but that excludes the single-exposure filters from the inputs.
• Use that detection image to find sources for the catalogs.
• Use the sources found in the detection image to do photometry on all the drizzled filter images (including the single-exposure images).

Finally, if every filter has only a single exposure (which would leave behind nothing to use for the detection image), all the filter images get included in the detection image.

[stscijgbot-hstdp]

Comment by Michele De La Pena on JIRA:

Rick White I have submitted a PR#⁠1797 for this issue. 

On the Linux systems, I have put the total detection image, as well as the associated catalogs for j8rh01 in /home/mdelapena/ForRick so you can see the current results. Interestingly, the point algorithm finds lots of "sources" in the nebulosity region, but presumably this is a different problem.

[stscijgbot-hstdp]

Comment by Rick White on JIRA:

That segment catalog looks very good to me (as does the total image). Here are some links to see various versions of the image: HAP image in ops New HAP image HLA version of image It is easier to see the catalog source positions if you use the Invert button to switch the color scale.

There are more artifacts around bright stars in the HAP catalog, but otherwise it is a very nice catalog.

The problem with the point catalog looks like the issue described in ticket HLA-1240. It is interesting that it only appeared after the cosmic rays were removed from the total image! I think this is likely to be due to some kind of problem with the background.