corrupted FITS file?

[botteon]

Hi,

I have this error

raise IOError("An error occurred while reading the FITS file")

when I press "c" in checkfactor. Also the calibrator images are not shown. It should be an issue related to astropy, has anybody seen this before?

The error is related to this FITS file L230423_SB000_uv.dppp_1248F906Et_121MHz.pre-cal_chunk8_1248FDB71t_3g.wsclean_image11-MFS-image.mask1 in the facetselfcal directory. I tried to use fitscheck on that file and it returns:

fitscheck L230423_SB000_uv.dppp_1248F906Et_121MHz.pre-cal_chunk8_1248FDB71t_3g.wsclean_image11-MFS-image.mask1 
WARNING: VerifyWarning: Error validating header for HDU #0 (note: Astropy uses zero-based indexing).
    Unparsable card (BZERO), fix it first with .verify('fix').
There may be extra bytes after the last HDU or the file is corrupted. [astropy.io.fits.hdu.hdulist]
EXCEPTION 'L230423_SB000_uv.dppp_1248F906Et_121MHz.pre-cal_chunk8_1248FDB71t_3g.wsclean_image11-MFS-image.mask1' .. Empty or corrupt FITS file
1 errors

I compared the output of fitscheck in CEP3 for an identical run and it gives:

fitscheck L230423_SB000_uv.dppp_1248F906Et_121MHz.pre-cal_chunk8_1248FDB71t_3g.wsclean_image11-MFS-image.mask1 
MISSING 'L230423_SB000_uv.dppp_1248F906Et_121MHz.pre-cal_chunk8_1248FDB71t_3g.wsclean_image11-MFS-image.mask1' .. Checksum not found in HDU #0
1 errors

In CEP3 astropy is 1.0 and pyfits is 2.4.0 In the node where I am running I tested astropy 2.0dev17810 and 1.3 and pyfits 3.4

Note that I also have the same error in initial-subtract before it crashes:

2017-02-17 08:40:19 ERROR node.lofar.python_plugin: An error occurred while reading the FITS file

[botteon]

Note that I can open the .mask1 file with casaviewer, but I have the following warnings

2017-02-20 13:22:41 WARN    FITSCoordinateUtil::fromFITSHeader  Zero or no rest frequency provided for velocity axis.
2017-02-20 13:22:41 WARN    FITSImage::getImageAttributes (file /var/rpmbuild/BUILD/casa-prerelease/casa-prerelease-4.7.0/casacore/images/Images/FITSImage.cc, line 720)    No proper coordinate system defined in FITS file. Using dummy linear system instead.
2017-02-20 13:22:41 WARN    WCCSAxisLabeller::setSpectralState (file /var/rpmbuild/BUILD/casa-prerelease/casa-prerelease-4.7.0/code/display/DisplayCanvas/WCCSAxisLabeller.cc, line 658)    Illegal spectral unit 
2017-02-20 13:22:41 WARN    WCCSAxisLabeller::setSpectralState (file /var/rpmbuild/BUILD/casa-prerelease/casa-prerelease-4.7.0/code/display/DisplayCanvas/WCCSAxisLabeller.cc, line 658)    Illegal spectral unit 
2017-02-20 13:22:41 WARN    WCCSAxisLabeller::setSpectralState (file /var/rpmbuild/BUILD/casa-prerelease/casa-prerelease-4.7.0/code/display/DisplayCanvas/WCCSAxisLabeller.cc, line 658)    Illegal spectral unit 
2017-02-20 13:22:42 WARN    WCCSAxisLabeller::setSpectralState (file /var/rpmbuild/BUILD/casa-prerelease/casa-prerelease-4.7.0/code/display/DisplayCanvas/WCCSAxisLabeller.cc, line 658)    Illegal spectral unit 
2017-02-20 13:22:42 WARN    WCCSAxisLabeller::setSpectralState (file /var/rpmbuild/BUILD/casa-prerelease/casa-prerelease-4.7.0/code/display/DisplayCanvas/WCCSAxisLabeller.cc, line 658)    Illegal spectral unit 
2017-02-20 13:22:42 WARN    WCCSAxisLabeller::setSpectralState (file /var/rpmbuild/BUILD/casa-prerelease/casa-prerelease-4.7.0/code/display/DisplayCanvas/WCCSAxisLabeller.cc, line 658)    Illegal spectral unit 

(and the x and y axis of the plot are in km!?). Something definitely went wrong when the .mask1 file was written.

[botteon]

Update: the problem occurs when the mask is converted to fits by the make_clean_mask.py script. Without the "-f fits" flag make_clean_mask.py produces a corrected (CASA-format) mask.

The conversion in make_clean_mask.py is done by the tofits module of python-casacore, perhaps @tammojan could you have a look in this? We are using v2.1.0. Thanks.

Alternatively, how to set factor to save automatically CASA format mask files?

[tammojan]

Sorry for missing this thread! I tried running make_clean_mask.py in isolation (on CEP3):

~/opt/factor/factor/scripts/make_clean_mask.py ./hdf5image/H227+48_BAND7.img.restored.fits output.fits
casaviewer output.fits

This works. Perhaps the issue is already present in the input image; if you can give me the input image I'd be happy to try it on CEP3.

[botteon]

Hey Tammo, thanks for your test. The file is

L230423_SB000_uv.dppp_1248F906Et_121MHz.pre-cal_chunk8_1248FDB71t_0g.wsclean_image11-image.fits.zip

I've already tested it on CEP3

python /home/rafferty/LOFAR/factor/factor/scripts/make_clean_mask.py -f fits L230423_SB000_uv.dppp_1248F906Et_121MHz.pre-cal_chunk8_1248FDB71t_0g.wsclean_image11-image.fits test_fits.mask.fits

and

python /home/rafferty/LOFAR/factor/factor/scripts/make_clean_mask.py -f casa L230423_SB000_uv.dppp_1248F906Et_121MHz.pre-cal_chunk8_1248FDB71t_0g.wsclean_image11-image.fits test_casa.mask

both produce a mask WITH coordinates.

I think there is a conflict with some libraries installed in the node in Bologna (the node is new and we installed the latest versions of everything). So, what are the libraries used in this conversion? We can compare Bologna vs CEP3 versions and see if it is an issue related to that. This is just an idea...

[tammojan]

Could you add a debug line to your Bologna instance of make_clean_mask.py: around line 499 (just before img.export_image) could you add a print image.header? This way we can see whether the coordinates have disappeared already before saving the image.

In your e-mail you referred to line 608 in make_clean_mask.py, but I don't think this line is executed.

[tammojan]

I think the actual conversion to fits is performed by PyBDSF's make_fits_image: https://github.com/lofar-astron/PyBDSF/blob/master/bdsf/functions.py#L1470

If I'm correct, this means that the error would be in the version of astropy rather than python-casacore.

[botteon]

Hi again, the headers seem both genuine. But let's do a step back. I noticed that using make_clean_mask.py with the following flags (same parameters used by factor):

-p 8 -a False -i 7 -r '(120,30)' -d float -b 0.4 -t True -o True -f fits

produces a mask WITHOUT coordinates. If i replace -f fits with -f casa the mask HAS coordinates.

Now, if I run make_clean_mask.py only with the -f flag the output file HAS the coordinates regardless the format. So I tried to add the flags one by one and in the end it seems that the problem is due to the region trim in combination with the fits format: -b 0.4 -f fits does NOT produce a mask with coordinate.

Please note that i did a test in CEP3 and the script with -b 0.4 -f fits works so, again, it should be a library conflict in the node in Bologna. We are using Astropy 1.3.2 (we also tried 1.3).

[tammojan]

Ok, thanks for the added parameters. In this case, the FITS file is written by https://github.com/lofar-astron/factor/blob/master/factor/scripts/make_clean_mask.py#L608 . Could you add just before line 608 a debug line with print new_mask.coordinates() ? This will test whether the coordinates are still in there just before converting to fits with python-casacore.

[botteon]

Hi, I've added the debug line.

If I run make_clean_mask.py -b 0.4 -f fits I obtain

Spectral Coordinate: Reference Pixel : 0.0 Reference Value : 124119567.871 Hz Increment : 300000000.0 Hz Frame : TOPO Rest Frequency : 124119567.871 Hz Stokes Coordinate: Reference Pixel : [ 0.] Reference Value : [ 1.] [] Increment : [ 1.] [] Direction Coordinate: Reference Pixel : [ 256. 256.] Reference Value : [ 0.86857995 -2.68128158] ['rad', 'rad'] Increment : [ 7.27220522e-06 -7.27220522e-06] ['rad', 'rad'] Frame : J2000 Projection : SIN

Note that I don't obtain the same output I if run -b 0.4 -f casa or only -f fits. It seems that the code read the debug line only with -b 0.4 -f fits.

[tammojan]

Note that I don't obtain the same output I if run -b 0.4 -f casa or only -f fits. It seems that the code read the debug line only with -b 0.4 -f fits.

That's probably because of the line if img_format == 'fits': just above it.

Looking at the output, indeed the error must be in python-casacore. I've made a ticket there, let's continue the discussion there.