Incomplete Header cards with the stream:// driver

[at88mph]

It would be really useful to be able to use:

import fitsio
from io import BytesIO
from sys import stdout

output = BytesIO()
fitsio.write(output)

# OR

fitsio.write(stdout)

I have a use case whereby I want to serve this in a Flask or Gunicorn server, and I want to stream the output of a slice to the Response. FITSIO supports writing to the hyphen (-), but I can't trap that I don't think.

Is this a trivial fix? It doesn't look like it, but my C->Python knowledge is rusty.

[at88mph]

I've started using CFITSIO's stream:// output and I think that will solve this. However, the documentation states that it's "fragile", and I'm wondering if it conforms to the CFITSIO documentation here: https://heasarc.gsfc.nasa.gov/fitsio/c/c_user/node88.html

The reason I ask is because writing some data fails with:

  File "/usr/local/lib/python3.7/site-packages/fitsio/fitslib.py", line 624, in write
    header=header)
  File "/usr/local/lib/python3.7/site-packages/fitsio/fitslib.py", line 680, in write_image
    compress=compress, tile_dims=tile_dims)
  File "/usr/local/lib/python3.7/site-packages/fitsio/fitslib.py", line 845, in create_image_hdu
    extver=extver)
OSError: FITSIO status = 108: error reading from FITS file
Error reading data buffer from file:
stream://
ffflus could not reopen the current HDU.

I suspect there may be a violation of points 3 or 4 from the bulleted list in the link above. Has anyone successfully used the stream:// to read or write?

[at88mph]

I have a business need to use the stream:// driver. I've moved this ticket to an issue with using it.

I've attached a test phu.fits.gz file.

import fitsio

input = fitsio.FITS('phu.fits.gz', 'r')
output = fitsio.FITS('stream://', 'rw')

hdu = input[0]
output.write(None, header=hdu.read_header())

Using Python 3.7.4 spits out a partial Header with an error:

SIMPLE  =                    T / file does conform to FITS standard             BITPIX  =                   16 / number of bits per data pixel                  NAXIS   =                    0 / number of data axes                            EXTEND  =                    T / FITS dataset may contain extensions            COMMENT   FITS (Flexible Image Transport System) format is defined in 'AstronomyCOMMENT   and Astrophysics', volume 376, page 359; bibcode: 2001A&A...376..359H                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                 Traceback (most recent call last):
  File "script.py", line 7, in <module>
    output.write(None, header=hdu.read_header())
  File "/usr/local/lib/python3.7/site-packages/fitsio/fitslib.py", line 545, in write
    header=header)
  File "/usr/local/lib/python3.7/site-packages/fitsio/fitslib.py", line 599, in write_image
    compress=compress, tile_dims=tile_dims)
  File "/usr/local/lib/python3.7/site-packages/fitsio/fitslib.py", line 763, in create_image_hdu
    extver=extver)
OSError: FITSIO status = 107: tried to move past end of file
Failed to find the END keyword in header (ffgphd).
Failed to read the required primary array header keywords.
ffflus could not reopen the current HDU.

If I run output.write(None, header=hdu.read_header()) a second time, it will spit out the rest of the header, so there may be a flushing issue somewhere.

I've tested directly with imcopy and it works with the stream:// driver without issue, so I think it's something in the C binding with create_image_hdu.

[dstndstn]

You can do:

compression = None # CFITSIO compression specifier, eg '[compress G]'
F = fitsio.FITS('mem://' + (compression or ''), 'rw')
# do your business with F
rawdata = F.read_raw()
F.close()

[esheldon]

@at88mph @dstndstn where do we stand on this one?

[dstndstn]

I've never used the 'stream' driver, but we use the 'mem' driver all the time (NERSC filesystem corruption grumble grumble)

[at88mph]

When I last investigated this I tracked it down to the C bindings, but never got a fix for it. The CFITSIO maintainers call it experimental, so maybe it's not useable yet.