search

opencadc / vlass2caom2

Application to generate a CAOM2 observation from VLASS FITS files.
0 stars 2 forks source link

VLASS file stored incorrectly at CADC #9

Open SharonGoliath opened 4 years ago

SharonGoliath commented 4 years ago

From CIRADA:

Hey JJ and Severin, I wanted to ask what the error checking procedure was for the NRAO VLASS -> CADC transfer was. We are having difficulty with a cutout request: https://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/caom2ops/sync?ID=ad%3AVLASS%2FVLASS1.1.ql.T03t25.J161126-293000.10.2048.v1.I.iter1.image.pbcor.tt0.rms.subim.fits&CIRCLE=243.17546003438235+-29.897624536684397+0.016666666666666666 (modifié) [10 h 54] We have tried changing the cutout region, or even downloading the whole thing, and we are seeing issues in the FITS files (DS9 is throwing SIGBUS and SIGEV errors, while astropy is giving warnings like: “File may have been truncated: actual file length (69120) is smaller than the expected size (72000) [astropy.io.fits.file]” [10 h 54] I’ve downloaded the original file from NRAO https://archive-new.nrao.edu/vlass/quicklook/VLASS1.1/T03t25/VLASS1.1.ql.T03t25.J161126-293000.10.2048.v1/VLASS1.1.ql.T03t25.J161126-293000.10.2048.v1.I.iter1.image.pbcor.tt0.rms.subim.fits and that file seems to be ok.

[10 h 55] The good news is that this may be a relatively rare error.

Falon [10 h 56] the error is related to astropy or ds9 trying to access info of an array that isn't there, but is told it should be. (why segfaults and the buffer issue from astropy happen sometimes related to what the header says being misaligned with the actual data)

SharonGoliath commented 4 years ago

Before and after re-retrieval:

vicadmins-MBP-2:gem2caom2 goliaths$ cadc-data info --cert $HOME/.ssl/cadcproxy.pem VLASS VLASS1.1.ql.T03t25.J161126-293000.10.2048.v1.I.iter1.image.pbcor.tt0.rms.subim.fits File VLASS1.1.ql.T03t25.J161126-293000.10.2048.v1.I.iter1.image.pbcor.tt0.rms.subim.fits: archive: VLASS encoding: None lastmod: Fri, 27 Jul 2018 03:10:21 GMT md5sum: 4d0eb488433c5c8fc8045d9e66b323c7 name: VLASS1.1.ql.T03t25.J161126-293000.10.2048.v1.I.iter1.image.pbcor.tt0.rms.subim.fits size: 7159808 type: application/fits umd5sum: 4d0eb488433c5c8fc8045d9e66b323c7 usize: 7159808 vicadmins-MBP-2:gem2caom2 goliaths$ cadc-data info --cert $HOME/.ssl/cadcproxy.pem VLASS VLASS1.1.ql.T03t25.J161126-293000.10.2048.v1.I.iter1.image.pbcor.tt0.rms.subim.fits File VLASS1.1.ql.T03t25.J161126-293000.10.2048.v1.I.iter1.image.pbcor.tt0.rms.subim.fits: archive: VLASS encoding: binary lastmod: Wed, 16 Sep 2020 19:54:49 GMT md5sum: edef46c27fa198dd746b1d9831519e7c name: VLASS1.1.ql.T03t25.J161126-293000.10.2048.v1.I.iter1.image.pbcor.tt0.rms.subim.fits size: 55425600 type: application/fits umd5sum: edef46c27fa198dd746b1d9831519e7c usize: 55425600

SharonGoliath commented 4 years ago

The verification code does an astropy.io.fits.open call, but it was added on May 28, 2019, so several months after the original retrieval. The second check is a getdata call, which was added Nov 18, 2019, when NEOSSAT files were failing in interesting ways.

The git commit time is not necessarily the vlass2caom2 execution time.

SharonGoliath commented 4 years ago

From JJ: I think we may have is to go back to all the files loaded before May 2020 and reload them. That would be the cleanest approach.