Closed KiaraSpin closed 6 years ago
Description of the problem: when @nivyaaroy runs 2dphot on KiDS DR1/DR2 images published in DR3, weird results are produced. The reason of that was tracked down to how the subsection science images are produced In the plot below, you can see (from upper left, counter-clockwise), sections of: the full coadd, the science and mask subsections that @nivyaaroy produced. The sections were matched in pixels: the circles show the same WCS position, you can see that there is no issue in coadd vs. mask, while there is a problem in the science subsection In the plot below I matched in WCS a larger area of coadd (right) vs. subsection (left). The circle now shows the position of the same source in both images, for reference. The weird thing is that in the subsection there is a region (in the left part) which DOES NOT appear in the coadd, and actually is a part of the coadded image translated to the first half of the image ! So, to summarize: there is an issue in the production of science subsections (masks are ok), as a consequence we are running 2dphot on the wrong pixels (and using mask informations from the right pixels). In addition, running listhead on the science subsection seems corrupted I get errors, which points to problems in the fits file. To try to understand the problem, @mariorpd13 selected from the script that @nivyaaroy is using to run 2dphot, only those lines that produce the subsections. This is done in thor, using the same KiDS image used for the above tests: /thornas2/tig/KIDS_ESO_DR3/KiDS_DR1.0129.0-0.5_r_sci.fits. The script is: /home/radovich/test_kids/test_n.x. However, I CAN'T reproduce the problem: the science subsection produced by THIS script is ok (no pixel mismatch, no error in the fits file). Since the subsection is produced with a modified version of the imhead code, it is possible that the issue is due to problems in the FITSIO library. so it should be checked if @nivyaaroy and @mariorpd13 are using the same versions of imarith/CFITSIO (@mariorpd13: /usr/local/bin/imarith, /home/radovich/pullecenella/program/cfitsio)
Dear Francesco, Fedor and Nivya,
Mario added all the information, please, can you check this issue?
C
On 17/04/2018 09:42, mariorpd13 wrote:
Description of the problem: when @nivyaaroy https://github.com/nivyaaroy runs 2dphot on KiDS DR1/DR2 images published in DR3, weird results are produced. The reason of that was tracked down to how the subsection science images are produced In the plot below, you can see (from upper left, counterclock-wise), sections of: the full coadd, the science and mask subsections that @nivyaaroy https://github.com/nivyaaroy produced. The sections were matched in pixels: the circles show the same WCS position, you can see that there is no iusse in coadd vs. mask, while there is a problem in the science subsection match_pixels https://user-images.githubusercontent.com/14972397/38854489-ea5a4880-4220-11e8-861d-e6c3889b5b48.png In the plot below I matched in WCS a larger area of coadd (left) vs. subsection (right). The weird thing is that in the subsection there is a region (in he left part) which DOES NOT appear in the coadd, and actually is a part of the coadded image translated to the first half of the image ! A match_wcs https://user-images.githubusercontent.com/14972397/38854497-ee5e0dfe-4220-11e8-8d66-f646456a9573.png So, to summarize: there is an issue in the production of science subsections (masks are ok), as a consequence we are running 2dphot on the wrong pixels (and using mask informations from the right pixels). In addition, running listhead on the science subsection seems corrupted I get errors, which points to problems in the fits file. To try to understand the problem, I selected from the script that @nivyaaroy https://github.com/nivyaaroy is using to run 2dphot, only those lines that produce the subsections. This is done in thor, using the same KiDS image used for the above tests: /thornas2/tig/KIDS_ESO_DR3/KiDS_DR1.0129.0-0.5_r_sci.fits. The script is: /home/radovich/test_kids/test_n.x. However, I CAN'T reproduce the problem: the science subsection produced by THIS script is ok (no pixel mismatch, no error in the fits file). Since the subsection is produced with a modified version of the imhead code, it is possible that the issue is due to problems in the FITSIO library. so it should be checked if @nivyaaroy https://github.com/nivyaaroy and @mariorpd13 https://github.com/mariorpd13 are using the same versions of imarith/CFITSIO (@mariorpd13 https://github.com/mariorpd13: /usr/local/bin/imarith, /home/radovich/pullecenella/program/cfitsio)
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/nrnapolita/KIWI_LSST/issues/18#issuecomment-381882273, or mute the thread https://github.com/notifications/unsubscribe-auth/AgUtQh5VJtCRCtmVP8ZshB_OD8FCFpSiks5tpZzggaJpZM4TWe0r.
-- Dr. Crescenzo Tortora
/Post-Doc/ Kapteyn Astronomical Institute Postbus 800, 9700 AV, Groningen The Netherlands
Tel: +31 (0)50-3634088 e-mail (1): ctortora@astro.rug.nl e-mail (2): ctortora@na.astro.it web-page: https://www.astro.rug.nl/~ctortora/ http://www.astro.rug.nl/%7Ectortora/
On thor all imarth programm is the same, compiled staticaly with version cfitsio 3.30. So, it really strange how wrong cutting was done.
Yes, it is weird: the only thing I can think is to ask Nivya to look at my script, check that we are doing the same thing on the same data, and try to run it to see what she gets.....
We are using the same data. I checked the code from Mario. Basically the commands are the same as in the code I used. Anyway I am trying to run the code from Mario too. I don't know whether it is the issue with the version or not. But why the code I used works for DR3 tiles and not for DR1 and DR2?
@nivyaaroy run the "simplified" script on her account: the subsection image that she gets is correct. In the picture: left: "old" subsection (/home/roy/check_intdr3/ESO_DR3/KiDS_DR1.0129.0-0.5/KiDS_DR1.0129.0-0.5_r_01_01.fits) right: "new" subsection (/home/roy/check_intdr3/ESO_DR3/KiDS_DR1.0129.0-0.5_code_mario/KiDS_DR1.0129.0-0.5_r_01_01_n.fits) So, the question is: why are the two different, since they are produced using the same input images and the same code/libraries ??
@mariorpd13, @nivyaaroy sorry I did miss some steps: Is the problem related to using two different CODES on the same images or the problem arises when running BOTH CODES on DR1 and DR2 images? i.e. is it a CODE-ISSUE or DATA-ISSUE?
No.... we are using the SAME code on the SAME image on the SAME machine.... So it seems more a... magic-issue ;-) I suggested Nivya to run again the full 2dphot script on that tile: maybe there was some temporary hardware/disk problem when she did that the first time, which corrupted the outputs (Fedor, is it possible... ?). It is a crazy idea, but I can't think of anything else.....
Generally speaking, it's not impossible (overlap in memory allocation, memory page reuse, wrong pointers and so on), but for my experience that quite improbable and can been happened one time, probability to have repeated the same hardware error is too small. More probable software problem.
I do not believe in magic... I believe in science :D Did you try to "share asd compare" the source codes? I mean... it could be that one of you changed a little tiny thing into the code... perhaps without realizing or remembering it?
By the way, this is the error that I get with listhead on the "wrong" subsection image (and on many but not all subsection images in the same folder):
listhead /home/roy/check_intdr3/ESO_DR3/KiDS_DR1.0129.0-0.5/KiDS_DR1.0129.0-0.5_r_01_01.fits
FITSIO status = 252: 1st key not SIMPLE or XTENSION Extension doesn't start with SIMPLE or XTENSION keyword. (ffrhdu) ���R@�;��ڮ��|�Oν�y�9����3����\7�Q�_���I>c��<��@��<��� ?M|h?�)�@:��@��~ Failed to move to HDU number 2 (ffmahd).
Problem solved !
What was the problem?
C
On 19/04/2018 16:51, mariorpd13 wrote:
Problem solved !
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/nrnapolita/KIWI_LSST/issues/18#issuecomment-382765769, or mute the thread https://github.com/notifications/unsubscribe-auth/AgUtQrEA72D_v87fC0BBMf8h_2GosViwks5tqKRtgaJpZM4TWe0r.
-- Dr. Crescenzo Tortora
/Post-Doc/ Kapteyn Astronomical Institute Postbus 800, 9700 AV, Groningen The Netherlands
Tel: +31 (0)50-3634088 e-mail (1): ctortora@astro.rug.nl e-mail (2): ctortora@na.astro.it web-page: https://www.astro.rug.nl/~ctortora/ http://www.astro.rug.nl/%7Ectortora/
I will never tell :-D
In the original script, after imarith there is a call to a delhead code, which is the bad guy who corrupted the fits file.... So the solution is just to comment it...
Mario
On 19/04/2018 16:52, ctortora wrote:
What was the problem?
C
On 19/04/2018 16:51, mariorpd13 wrote:
Problem solved !
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub
https://github.com/nrnapolita/KIWI_LSST/issues/18#issuecomment-382765769,
or mute the thread
-- Dr. Crescenzo Tortora
/Post-Doc/ Kapteyn Astronomical Institute Postbus 800, 9700 AV, Groningen The Netherlands
Tel: +31 (0)50-3634088 e-mail (1): ctortora@astro.rug.nl e-mail (2): ctortora@na.astro.it web-page: https://www.astro.rug.nl/~ctortora/ http://www.astro.rug.nl/%7Ectortora/
— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/nrnapolita/KIWI_LSST/issues/18#issuecomment-382766236, or mute the thread https://github.com/notifications/unsubscribe-auth/AOR17Q5J2fqGqGCrRClSSovfB5uR3kHBks5tqKS0gaJpZM4TWe0r.
AI @mariorpd13 -> can you please collect here all the problems and the checks that you did concerning this issue? We need a complete description of the problem including what has to be done and by whom?