spacetelescope / stdatamodels

https://stdatamodels.readthedocs.io
Other
5 stars 24 forks source link

Assigning `f8` 2d data to a table can lead to corrupt output #280

Open braingram opened 4 months ago

braingram commented 4 months ago

Let's assume we have a schema for TableModel that contains

      one_column_table:
        fits_hdu: ONECOL
        datatype:
          - name: col
            datatype: float64

If we assign data to the model as follows, the produced file is corrupt:

fn = tmp_path / "test.fits"

write_model = TableModel()
arr = np.arange(16 * 32, dtype='f8').reshape((16, 32))
write_model.one_column_table = arr
np.testing.assert_equal(arr, write_model.one_column_table['col'])
write_model.save(fn)

with TableModel(fn) as read_model:
    np.testing.assert_equal(arr, read_model.one_column_table['col'])

The above code fails on TableModel(fn) with:

E           astropy.utils.exceptions.AstropyUserWarning: ERROR loading embedded ASDF: non-ASCII characters are present in the FITS file header and have been replaced by "?" characters

This can be further reduced to just astropy:

from astropy.io import fits
import numpy as np

hdulist = fits.HDUList()
hdulist.append(fits.PrimaryHDU())

arr = np.arange(32 * 16).reshape((32, 16)).astype(dtype=[('col', 'f8')])

hdulist.append(fits.BinTableHDU())
hdulist[-1].data = arr

hdulist.writeto('foo.fits', overwrite=True)

with fits.open('foo.fits') as ff:
    ff.info()

Running the above code results in:

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 2: ordinal not in range(128)
and
OSError: Header missing END card.

Note that in all cases the error doesn't occur until the data is read. No errors are issued on write.

braingram commented 4 months ago

Using f4 does not produce the same error but does produce a file with an incorrectly shaped array ((32) instead of (32, 16)).

braingram commented 4 months ago

This may be related to https://github.com/spacetelescope/stdatamodels/issues/215