nrnapolita / KIWI_LSST

0 stars 0 forks source link

Issues related to different DR in KIDS (sizes, number density, ...) #18

Closed KiaraSpin closed 6 years ago

KiaraSpin commented 6 years ago

AI @mariorpd13 -> can you please collect here all the problems and the checks that you did concerning this issue? We need a complete description of the problem including what has to be done and by whom?

mariorpd13 commented 6 years ago

Description of the problem: when @nivyaaroy runs 2dphot on KiDS DR1/DR2 images published in DR3, weird results are produced. The reason of that was tracked down to how the subsection science images are produced In the plot below, you can see (from upper left, counter-clockwise), sections of: the full coadd, the science and mask subsections that @nivyaaroy produced. The sections were matched in pixels: the circles show the same WCS position, you can see that there is no issue in coadd vs. mask, while there is a problem in the science subsection match_pixels In the plot below I matched in WCS a larger area of coadd (right) vs. subsection (left). The circle now shows the position of the same source in both images, for reference. The weird thing is that in the subsection there is a region (in the left part) which DOES NOT appear in the coadd, and actually is a part of the coadded image translated to the first half of the image ! match_wcs So, to summarize: there is an issue in the production of science subsections (masks are ok), as a consequence we are running 2dphot on the wrong pixels (and using mask informations from the right pixels). In addition, running listhead on the science subsection seems corrupted I get errors, which points to problems in the fits file. To try to understand the problem, @mariorpd13 selected from the script that @nivyaaroy is using to run 2dphot, only those lines that produce the subsections. This is done in thor, using the same KiDS image used for the above tests: /thornas2/tig/KIDS_ESO_DR3/KiDS_DR1.0129.0-0.5_r_sci.fits. The script is: /home/radovich/test_kids/test_n.x. However, I CAN'T reproduce the problem: the science subsection produced by THIS script is ok (no pixel mismatch, no error in the fits file). Since the subsection is produced with a modified version of the imhead code, it is possible that the issue is due to problems in the FITSIO library. so it should be checked if @nivyaaroy and @mariorpd13 are using the same versions of imarith/CFITSIO (@mariorpd13: /usr/local/bin/imarith, /home/radovich/pullecenella/program/cfitsio)

ctortora commented 6 years ago

Dear Francesco, Fedor and Nivya,

Mario added all the information, please, can you check this issue?

C

On 17/04/2018 09:42, mariorpd13 wrote:

Description of the problem: when @nivyaaroy https://github.com/nivyaaroy runs 2dphot on KiDS DR1/DR2 images published in DR3, weird results are produced. The reason of that was tracked down to how the subsection science images are produced In the plot below, you can see (from upper left, counterclock-wise), sections of: the full coadd, the science and mask subsections that @nivyaaroy https://github.com/nivyaaroy produced. The sections were matched in pixels: the circles show the same WCS position, you can see that there is no iusse in coadd vs. mask, while there is a problem in the science subsection match_pixels https://user-images.githubusercontent.com/14972397/38854489-ea5a4880-4220-11e8-861d-e6c3889b5b48.png In the plot below I matched in WCS a larger area of coadd (left) vs. subsection (right). The weird thing is that in the subsection there is a region (in he left part) which DOES NOT appear in the coadd, and actually is a part of the coadded image translated to the first half of the image ! A match_wcs https://user-images.githubusercontent.com/14972397/38854497-ee5e0dfe-4220-11e8-8d66-f646456a9573.png So, to summarize: there is an issue in the production of science subsections (masks are ok), as a consequence we are running 2dphot on the wrong pixels (and using mask informations from the right pixels). In addition, running listhead on the science subsection seems corrupted I get errors, which points to problems in the fits file. To try to understand the problem, I selected from the script that @nivyaaroy https://github.com/nivyaaroy is using to run 2dphot, only those lines that produce the subsections. This is done in thor, using the same KiDS image used for the above tests: /thornas2/tig/KIDS_ESO_DR3/KiDS_DR1.0129.0-0.5_r_sci.fits. The script is: /home/radovich/test_kids/test_n.x. However, I CAN'T reproduce the problem: the science subsection produced by THIS script is ok (no pixel mismatch, no error in the fits file). Since the subsection is produced with a modified version of the imhead code, it is possible that the issue is due to problems in the FITSIO library. so it should be checked if @nivyaaroy https://github.com/nivyaaroy and @mariorpd13 https://github.com/mariorpd13 are using the same versions of imarith/CFITSIO (@mariorpd13 https://github.com/mariorpd13: /usr/local/bin/imarith, /home/radovich/pullecenella/program/cfitsio)

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/nrnapolita/KIWI_LSST/issues/18#issuecomment-381882273, or mute the thread https://github.com/notifications/unsubscribe-auth/AgUtQh5VJtCRCtmVP8ZshB_OD8FCFpSiks5tpZzggaJpZM4TWe0r.

-- Dr. Crescenzo Tortora

/Post-Doc/ Kapteyn Astronomical Institute Postbus 800, 9700 AV, Groningen The Netherlands

Tel: +31 (0)50-3634088 e-mail (1): ctortora@astro.rug.nl e-mail (2): ctortora@na.astro.it web-page: https://www.astro.rug.nl/~ctortora/ http://www.astro.rug.nl/%7Ectortora/

FedorGetman commented 6 years ago

On thor all imarth programm is the same, compiled staticaly with version cfitsio 3.30. So, it really strange how wrong cutting was done.

mariorpd13 commented 6 years ago

Yes, it is weird: the only thing I can think is to ask Nivya to look at my script, check that we are doing the same thing on the same data, and try to run it to see what she gets.....

nivyaaroy commented 6 years ago

We are using the same data. I checked the code from Mario. Basically the commands are the same as in the code I used. Anyway I am trying to run the code from Mario too. I don't know whether it is the issue with the version or not. But why the code I used works for DR3 tiles and not for DR1 and DR2?

mariorpd13 commented 6 years ago

@nivyaaroy run the "simplified" script on her account: the subsection image that she gets is correct. In the picture: left: "old" subsection (/home/roy/check_intdr3/ESO_DR3/KiDS_DR1.0129.0-0.5/KiDS_DR1.0129.0-0.5_r_01_01.fits) right: "new" subsection (/home/roy/check_intdr3/ESO_DR3/KiDS_DR1.0129.0-0.5_code_mario/KiDS_DR1.0129.0-0.5_r_01_01_n.fits) images_nivya So, the question is: why are the two different, since they are produced using the same input images and the same code/libraries ??

KiaraSpin commented 6 years ago

@mariorpd13, @nivyaaroy sorry I did miss some steps: Is the problem related to using two different CODES on the same images or the problem arises when running BOTH CODES on DR1 and DR2 images? i.e. is it a CODE-ISSUE or DATA-ISSUE?

mariorpd13 commented 6 years ago

No.... we are using the SAME code on the SAME image on the SAME machine.... So it seems more a... magic-issue ;-) I suggested Nivya to run again the full 2dphot script on that tile: maybe there was some temporary hardware/disk problem when she did that the first time, which corrupted the outputs (Fedor, is it possible... ?). It is a crazy idea, but I can't think of anything else.....

FedorGetman commented 6 years ago

Generally speaking, it's not impossible (overlap in memory allocation, memory page reuse, wrong pointers and so on), but for my experience that quite improbable and can been happened one time, probability to have repeated the same hardware error is too small. More probable software problem.

KiaraSpin commented 6 years ago

I do not believe in magic... I believe in science :D Did you try to "share asd compare" the source codes? I mean... it could be that one of you changed a little tiny thing into the code... perhaps without realizing or remembering it?

mariorpd13 commented 6 years ago

By the way, this is the error that I get with listhead on the "wrong" subsection image (and on many but not all subsection images in the same folder):

listhead /home/roy/check_intdr3/ESO_DR3/KiDS_DR1.0129.0-0.5/KiDS_DR1.0129.0-0.5_r_01_01.fits

FITSIO status = 252: 1st key not SIMPLE or XTENSION Extension doesn't start with SIMPLE or XTENSION keyword. (ffrhdu) ���R@�;��ڮ��|�Oν�y�9����3����\7�Q�_���I>c��<��@��<��� ?M|h?�)�@:��@��~ Failed to move to HDU number 2 (ffmahd).

mariorpd13 commented 6 years ago

Problem solved !

ctortora commented 6 years ago

What was the problem?

C

On 19/04/2018 16:51, mariorpd13 wrote:

Problem solved !

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/nrnapolita/KIWI_LSST/issues/18#issuecomment-382765769, or mute the thread https://github.com/notifications/unsubscribe-auth/AgUtQrEA72D_v87fC0BBMf8h_2GosViwks5tqKRtgaJpZM4TWe0r.

-- Dr. Crescenzo Tortora

/Post-Doc/ Kapteyn Astronomical Institute Postbus 800, 9700 AV, Groningen The Netherlands

Tel: +31 (0)50-3634088 e-mail (1): ctortora@astro.rug.nl e-mail (2): ctortora@na.astro.it web-page: https://www.astro.rug.nl/~ctortora/ http://www.astro.rug.nl/%7Ectortora/

mariorpd13 commented 6 years ago

I will never tell :-D

In the original script, after imarith  there is a call to a delhead code, which is the bad guy who corrupted the fits file.... So the solution is just to comment it...

   Mario

On 19/04/2018 16:52, ctortora wrote:

What was the problem?

C

On 19/04/2018 16:51, mariorpd13 wrote:

Problem solved !

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub

https://github.com/nrnapolita/KIWI_LSST/issues/18#issuecomment-382765769,

or mute the thread

https://github.com/notifications/unsubscribe-auth/AgUtQrEA72D_v87fC0BBMf8h_2GosViwks5tqKRtgaJpZM4TWe0r.

-- Dr. Crescenzo Tortora

/Post-Doc/ Kapteyn Astronomical Institute Postbus 800, 9700 AV, Groningen The Netherlands

Tel: +31 (0)50-3634088 e-mail (1): ctortora@astro.rug.nl e-mail (2): ctortora@na.astro.it web-page: https://www.astro.rug.nl/~ctortora/ http://www.astro.rug.nl/%7Ectortora/

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/nrnapolita/KIWI_LSST/issues/18#issuecomment-382766236, or mute the thread https://github.com/notifications/unsubscribe-auth/AOR17Q5J2fqGqGCrRClSSovfB5uR3kHBks5tqKS0gaJpZM4TWe0r.