Undetected sources in proc_Images

[YarinMS]

Following the pipeline products inspection, we noticed sources which are undetected by the pipeline. While most of the undetected sources appear next to other stars, there are some isolated sources which haven't been detected as well. This happens in every image and in every subframe we examined. I attach few examples,

in/last10e/data1/archive/LAST.01.10.01/2023/12/14/proc/233328v0/LAST.01.10.01_20231214.233628.186_clear_074+55_010_001_009_sci_proc_Image_1.fits

SOME of the undetected sources x,y pixels : (X,Y) = (649,662), (1308,775), (1255,800), (1651,1029), (1548,1056), (1572,1214)

in /last10e/data1/archive/LAST.01.10.01/2023/12/14/proc/233328v0/LAST.01.10.01_20231214.233628.186_clear_074+55_010_001_009_sci_proc_Image_1.fits

SOME of the undetected sources x,y pixels : (X,Y) = (803,1306), (938,1282), (60,406), (159,400), (175,477)

[EastEriq]

I also see unexpectedly clear but undetected sources in PTF images, and was looking into it as well. I'm not yet sure, but I think that has to do with the computation of the local variance of the image.

[EranOfek]

This is likely an artifact of the of the background/variance estimation. Two possibilities to look into:

1. local biases in the estimation that depend on nearby stars.
2. After the grid of background/variance is created it is interpolated to the entire image. The interpolation scheme is not perfect and may generate artifacts.

[EranOfek]

The main problem is that these sources are found, but then removed by the RemoveBadSources option. The source of the problem is in the imProc.sources.cleanSources function. For example, binary stars were removed due to a large shift between the PEAK position and the 1st moment position. Changed default max shift from 1 to 1.5 pix. Some other stars were removed due to large diff. between the annulus back and image back. Revising the cleanSources function.

[EastEriq]

I beg to differ. In this case for instance the clear source at the center of the image (circled in red, since it is in the GAIA spectral catalog, but without a yellow marker, denoting the result of imProc.sources.findMeasureSources) is not found even with RemoveBadSources=false. which obviously marks a lot of other junk. (I'm referring to the upper star, not that in saturation below it, which is tagged as saturated anyway)

Here is a zoom in to the values of image, computed background and variance, in a 60x60 neighborhood The background is produced by imProc.background.background(AI, 'SubSizeXY',[32,32],'Overlap',16).

The image is the second subimage in /raid/enrico/PTF/2ndTake/images/level1/proc/2016/11/10/f14/c4/p5/v1/PTF_201611105117_i_p_scie_t121654_u030037655_f14_p001793_c04.fits on euler, and we are talking of the source at coordinates ~[1820,1754].

[EranOfek]

Made the following changes

cleanSources main taks in now done by two new functions: imProc.mask.maskCR and imProc.mask.xpeak_x1_diff

By default sources rejection based on S/N measurement in annulus is not operated. Seems better - please keep an eye on this. [dev1 7bf66169]

[EastEriq]

No improvement observed with your latest commit.

On this particular image, I spotted already 9 cases like that, and counting.

[EranOfek]

This is likely because your BackSizeXY is too small. In this case, the back and var are overestimated due to the fact that most of the region of the background sub image is dominated by the star. Try to use [128 by 128]

[EastEriq]

Been there, done that, no way. With larger windows more sources are missed. The following is with [256,256]. Note that not only the three sources circled in red are not identified, but also two other weaker ones but yet clear, above the middle one.

[EastEriq]

In fact it is even the opposite. Background computation on [16,16] does capture the central source: I guess we'll have to descend the bowels of how the peak is identified.

[EranOfek]

1. findMeasureSources creating mask if doens't exist [dev1 d18f76fb]
2. I can't reproduce the PTF Halpha results. In fact using the default parameters: AI=imProc.sources.findMeasureSources(AI,'RemoveEdgeDist',5) I get very nice results including CR removal. If you like to see the results without CR removal use: AI=imProc.sources.findMeasureSources(AI,'RemoveEdgeDist',5,'FlagCR',false)

Closing. Please check and reopen if needed.

[EastEriq]

Well, no, the latest addition doesn't help in my reference case of PTF_201611105117_i_p_scie_t121654_u030037655_f14_p001793_c04.fits subimage 2.

I have been somewhat more systematic in my tests though. I now compiled a list of 13 problematic sources, and I made a script for testing how many of them are correctly found with this or that analysis parameter. 9 of them are present in the GAIA spectral catalog, and 4 not, but yet are very clear on the ds9 image (sometimes close to other source, still separated).

The result is:

• many sources were excluded because my imProc.mask.maskTracks() marked them erroneously as 'Streaks', which I used to filter out with imProc.sources.findMeasureSources(...,'BadBitNames'). I can comment out my call to imProc.mask.maskTracks() because the masking is already done in the PTF original, or unfilter the sources with bit Streaks on.
• Furthermore, a few sources are flagged as 'ColumnHigh' by my imProc.mask.maskBadColumns(), and I was excluding them as well. For this test image, maybe I can skip thid test because it is not crucial, but worser images may benefit from it.
• Still, at optimum I detect only 10 out of these 13 sources. The size upon which the background is computed does not look anymore critical, the result is 10/13 if I use [256,256] as much as 128 or 64 or 32. (empty, [8,8] or [16,16] give a few less).

Here are the images of the three still undetected sources of my list:

[1115,900] (in GAIA)

[1309,615] not in GAIA

[1037,1535] not in GAIA

[EranOfek]

Is this your ref image: /raid/enrico/PTF/2ndTake/images/level1/proc/2016/11/10/f14/c4/p5/v1/PTF_201611105117_i_p_scie_t121654_u030037655_f14_p001793_c04.fits

Inspecting this image at the corrdinates you listed suggest that this is a different field. Are these coordinates in the cropped image?

In any case, running with default parameters:

AI=imProc.background.background(AI);
AI=imProc.sources.findMeasureSources(AI)

I am happy with the results.

[EastEriq]

Maybe you missed that I am referring to subimage 2? Yes, the coordinates are those of the cropped image.