desihub / desispec

DESI spectral pipeline
BSD 3-Clause "New" or "Revised" License
36 stars 24 forks source link

fibermaps don't pass fitsverify #961

Open sbailey opened 4 years ago

sbailey commented 4 years ago

Our fibermap extensions don't pass fitsverify, e.g.

$> cd /global/cfs/cdirs/desi/spectro/redux/minisv2
$> fitsverify calibnight/20200219/fiberflatnight-r0-20200219.fits
...
=================== HDU 6: BINARY Table ====================

*** Error:   Keyword #157, BUNIT is not allowed in the Bin/ASCII table.

 160 header keywords

 FIBERMAP  (73 columns x 500 rows)
...

$> fitsverify exposures/20200219/00050986/cframe-b0-00050986.fits
...
=================== HDU 6: BINARY Table ====================

*** Error:   Keyword #300, SIMPLE is not allowed in extensions.
*** Error:   Keyword #301, EXTEND is not allowed in extensions.
*** Error:   Keyword #300, SIMPLE is duplicated or out of order.
*** Warning: Keyword #331, EPOCH is deprecated. Use EQUINOX instead.

 554 header keywords

 FIBERMAP  (131 columns x 500 rows)
...

These appear to be cosmetic issues about following the standard, but we should fix them to make it easier to use fitsverify to validate that none of the files in a production are corrupted.

weaverba137 commented 4 years ago

Ah, these are not merely cosmetic errors. SIMPLE should definitely not be in any header but the primary.

BUNIT only applies to image HDUs. For a binary table, individual columns have TUNITXX headers, but BUNIT makes no sense and is definitely an error.

sbailey commented 4 years ago

Yes, it is an error in the sense that SIMPLE and BUNIT shouldn't be there (it is a leftover from inheriting header keywords from some upstream file), but it is "cosmetic" in the sense that it doesn't prevent fitsio / astropy.io.fits / desispec.io.read_frame etc. from reading and using the file just fine. But they do prevent us from using fitsverify as a boolean check on whether the file is corrupted (as opposed to their current state of useable while not completely following the fits standard).

weaverba137 commented 4 years ago

OK, fair enough. Different definitions of "error"!