Closed YarinMS closed 7 months ago
I also see unexpectedly clear but undetected sources in PTF images, and was looking into it as well. I'm not yet sure, but I think that has to do with the computation of the local variance of the image.
This is likely an artifact of the of the background/variance estimation. Two possibilities to look into:
The main problem is that these sources are found, but then removed by the RemoveBadSources option. The source of the problem is in the imProc.sources.cleanSources function. For example, binary stars were removed due to a large shift between the PEAK position and the 1st moment position. Changed default max shift from 1 to 1.5 pix. Some other stars were removed due to large diff. between the annulus back and image back. Revising the cleanSources function.
I beg to differ. In this case for instance the clear source at the center of the image (circled in red, since it is in the GAIA spectral catalog, but without a yellow marker, denoting the result of imProc.sources.findMeasureSources
)
is not found even with
RemoveBadSources=false
.
which obviously marks a lot of other junk.
(I'm referring to the upper star, not that in saturation below it, which is tagged as saturated anyway)
Here is a zoom in to the values of image, computed background and variance, in a 60x60 neighborhood
The background is produced by
imProc.background.background(AI, 'SubSizeXY',[32,32],'Overlap',16)
.
The image is the second subimage in /raid/enrico/PTF/2ndTake/images/level1/proc/2016/11/10/f14/c4/p5/v1/PTF_201611105117_i_p_scie_t121654_u030037655_f14_p001793_c04.fits
on euler, and we are talking of the source at coordinates ~[1820,1754].
Made the following changes
cleanSources main taks in now done by two new functions: imProc.mask.maskCR and imProc.mask.xpeak_x1_diff
By default sources rejection based on S/N measurement in annulus is not operated. Seems better - please keep an eye on this. [dev1 7bf66169]
No improvement observed with your latest commit.
On this particular image, I spotted already 9 cases like that, and counting.
This is likely because your BackSizeXY is too small. In this case, the back and var are overestimated due to the fact that most of the region of the background sub image is dominated by the star. Try to use [128 by 128]
Been there, done that, no way. With larger windows more sources are missed. The following is with [256,256]. Note that not only the three sources circled in red are not identified, but also two other weaker ones but yet clear, above the middle one.
In fact it is even the opposite. Background computation on [16,16] does capture the central source:
I guess we'll have to descend the bowels of how the peak is identified.
Closing. Please check and reopen if needed.
Well, no, the latest addition doesn't help in my reference case of PTF_201611105117_i_p_scie_t121654_u030037655_f14_p001793_c04.fits
subimage 2.
I have been somewhat more systematic in my tests though. I now compiled a list of 13 problematic sources, and I made a script for testing how many of them are correctly found with this or that analysis parameter. 9 of them are present in the GAIA spectral catalog, and 4 not, but yet are very clear on the ds9 image (sometimes close to other source, still separated).
The result is:
imProc.mask.maskTracks()
marked them erroneously as 'Streaks'
, which I used to filter out with imProc.sources.findMeasureSources(...,'BadBitNames')
. I can comment out my call to imProc.mask.maskTracks()
because the masking is already done in the PTF original, or unfilter the sources with bit Streaks
on.'ColumnHigh'
by my imProc.mask.maskBadColumns()
, and I was excluding them as well. For this test image, maybe I can skip thid test because it is not crucial, but worser images may benefit from it. Here are the images of the three still undetected sources of my list:
[1115,900] (in GAIA)
[1309,615] not in GAIA
[1037,1535] not in GAIA
Is this your ref image: /raid/enrico/PTF/2ndTake/images/level1/proc/2016/11/10/f14/c4/p5/v1/PTF_201611105117_i_p_scie_t121654_u030037655_f14_p001793_c04.fits
Inspecting this image at the corrdinates you listed suggest that this is a different field. Are these coordinates in the cropped image?
In any case, running with default parameters:
AI=imProc.background.background(AI);
AI=imProc.sources.findMeasureSources(AI)
I am happy with the results.
Maybe you missed that I am referring to subimage 2? Yes, the coordinates are those of the cropped image.
Following the pipeline products inspection, we noticed sources which are undetected by the pipeline. While most of the undetected sources appear next to other stars, there are some isolated sources which haven't been detected as well. This happens in every image and in every subframe we examined. I attach few examples,
in
/last10e/data1/archive/LAST.01.10.01/2023/12/14/proc/233328v0/LAST.01.10.01_20231214.233628.186_clear_074+55_010_001_009_sci_proc_Image_1.fits
SOME of the undetected sources x,y pixels :
(X,Y) = (649,662), (1308,775), (1255,800), (1651,1029), (1548,1056), (1572,1214)
in
/last10e/data1/archive/LAST.01.10.01/2023/12/14/proc/233328v0/LAST.01.10.01_20231214.233628.186_clear_074+55_010_001_009_sci_proc_Image_1.fits
SOME of the undetected sources x,y pixels :
(X,Y) = (803,1306), (938,1282), (60,406), (159,400), (175,477)