BUG: sherpa.astro.ui.save_table fails with crates backend: "Unable to create column HEADER: unsupported datatype ('numpy.object_')"

[DougBurke]

Using CIAO 4.7 (OS-X), with Sherpa started up in the top-level of the code distribution (for easy access to a PHA test file):

% sherpa
-----------------------------------------------------
Welcome to Sherpa: CXC's Modeling and Fitting Package
-----------------------------------------------------
CIAO 4.7 Sherpa version 1 Thursday, December 4, 2014

sherpa-1> import sys
sherpa-2> sys.tracebacklimit = 10
sherpa-3> load_pha('sherpa/astro/datastack/tests/data/3c273.pi')
WARNING: systematic errors were not found in file 'sherpa/astro/datastack/tests/data/3c273.pi'
statistical errors were found in file 'sherpa/astro/datastack/tests/data/3c273.pi'
but not used; to use them, re-read with use_errors=True
read ARF file sherpa/astro/datastack/tests/data/3c273.arf
read RMF file sherpa/astro/datastack/tests/data/3c273.rmf
WARNING: systematic errors were not found in file 'sherpa/astro/datastack/tests/data/3c273_bg.pi'
statistical errors were found in file 'sherpa/astro/datastack/tests/data/3c273_bg.pi'
but not used; to use them, re-read with use_errors=True
read background file sherpa/astro/datastack/tests/data/3c273_bg.pi
sherpa-4> save_table(1, '/tmp/tbl.fits', ascii=False, clobber=True)
  File "/export/local/ciao-4.7/ots/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2883, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-6a29ee71142f>", line 1, in <module>
    save_table(1, '/tmp/tbl.fits', ascii=False, clobber=True)
  File "<string>", line 1, in save_table
  File "/export/local/ciao-4.7/lib/python2.7/site-packages/sherpa/astro/ui/utils.py", line 3162, in save_table
    ascii, clobber)
  File "/export/local/ciao-4.7/lib/python2.7/site-packages/sherpa/astro/io/__init__.py", line 391, in write_table
    backend.set_table_data(filename, data, names, ascii=ascii, clobber=clobber)
  File "/export/local/ciao-4.7/lib/python2.7/site-packages/sherpa/astro/io/crates_backend.py", line 1230, in set_table_data
    tbl.write(filename, clobber=True)
  File "/export/local/ciao-4.7/lib/python2.7/site-packages/pycrates/tablecrate.py", line 299, in write
    backend.write( self, outfile=outfile )
  File "/export/local/ciao-4.7/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 652, in write
    self.__write_block( crate, close_flag )
  File "/export/local/ciao-4.7/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 695, in __write_block
    self.__write_data(block, crate)
  File "/export/local/ciao-4.7/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 805, in __write_data
    col_dd = self.__create_column_descriptor( block, col )
ValueError: Unable to create column HEADER: unsupported datatype ('numpy.object_').
sherpa-5>

[DougBurke]

The problem is because sherpa.astro.ui.save_table calls sherpa.astro.io.write_data with the output of get_data - i.e. the DataPHA object in this case - as the second argument. Now write_data calls sherpa.astro.io._pack_table on this object, to get at the column names to create in the output file:

sherpa-6> d = get_data() sherpa-7> d._fields

('name', 'channel', 'counts', 'staterror', 'syserror', 'bin_lo', 'bin_hi', 'grouping', 'quality', 'exposure', 'backscal', 'areascal', 'header')

The problem here is the header field, since it's not a column, but a Meta object. It is stuck into a CrateData object, which complains on write out, as crates doesn't know how to serialize a Python object.

The quick and easy solution would be just to avoid the header field when converting, since sherpa.astro.io.crates_backend.set_table_data already skips the name field. However, this is a generic routine, so this is not a general-purpose solution.

One idea I'm thinking about is having some sort of data serialization method in the Data objects, which would separate out "data" (tabular or nD or ...) from "metadata" from "this is in the Data class but shouldn't be serialized". However, this is likely to take time to plan, design, and then add to the code base. It also may not turn out to be a sensible idea.

[DougBurke]

Just to note that the current CIAO 4.8 development version fails in a slightly different place, but I'm fairly sure it's the same basic problem (trying to serialize a Python object as a column). This was done with a CIAO build which has not yet picked up #77, which is why there's still a FutureWarning about None comparison.

/home/dburke/local/ciao-4.8/lib/python2.7/site-packages/sherpa/astro/io/crates_backend.py:1224: FutureWarning: comparison to `None` will result in an elementwise object comparison in the future.
  col_names = [name for name in col_names if data[name] != None]
Traceback (most recent call last):
  File "conv.py", line 8, in <module>
    ui.save_table(1, 'tbl.out', ascii=False, clobber=True)
  File "<string>", line 1, in save_table
  File "/home/dburke/local/ciao-4.8/lib/python2.7/site-packages/sherpa/astro/ui/utils.py", line 4590, in save_table
    ascii, clobber)
  File "/home/dburke/local/ciao-4.8/lib/python2.7/site-packages/sherpa/astro/io/__init__.py", line 391, in write_table
    backend.set_table_data(filename, data, names, ascii=ascii, clobber=clobber)
  File "/home/dburke/local/ciao-4.8/lib/python2.7/site-packages/sherpa/astro/io/crates_backend.py", line 1230, in set_table_data
    _set_column(tbl, name, data[name])
  File "/home/dburke/local/ciao-4.8/lib/python2.7/site-packages/sherpa/astro/io/crates_backend.py", line 179, in _set_column
    col.values = numpy.array(val)
  File "/home/dburke/local/ciao-4.8/lib/python2.7/site-packages/pycrates/cratedata.py", line 113, in _set_values
    arrlen = invals[ii].size
AttributeError: 'Meta' object has no attribute 'size'

[DougBurke]

With the current master branch (or close to it), we get a slightly-different error:

sherpa In [4]: sys.tracebacklimit = 10                                          

sherpa In [5]: save_table(1, 'delme.fits', ascii=False, clobber=True)           
  File "/proj/xena/ciaox_install/Linux.190930/ciao-4.12/ots/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-5-79c07fe6bab2>", line 1, in <module>
    save_table(1, 'delme.fits', ascii=False, clobber=True)
  File "<string>", line 1, in save_table
  File "/proj/xena/ciaox_install/Linux.190930/ciao-4.12/lib/python3.5/site-packages/sherpa-4.11.1+29.gd1a1966a-py3.5-linux-x86_64.egg/sherpa/astro/ui/utils.py", line 4856, in save_table
    ascii, clobber)
  File "/proj/xena/ciaox_install/Linux.190930/ciao-4.12/lib/python3.5/site-packages/sherpa-4.11.1+29.gd1a1966a-py3.5-linux-x86_64.egg/sherpa/astro/io/__init__.py", line 745, in write_table
    backend.set_table_data(filename, data, names, ascii=ascii, clobber=clobber)
  File "/proj/xena/ciaox_install/Linux.190930/ciao-4.12/lib/python3.5/site-packages/sherpa-4.11.1+29.gd1a1966a-py3.5-linux-x86_64.egg/sherpa/astro/io/crates_backend.py", line 1248, in set_table_data
    _set_column(tbl, name, data[name])
  File "/proj/xena/ciaox_install/Linux.190930/ciao-4.12/lib/python3.5/site-packages/sherpa-4.11.1+29.gd1a1966a-py3.5-linux-x86_64.egg/sherpa/astro/io/crates_backend.py", line 190, in _set_column
    col.values = numpy.array(val)
  File "/proj/xena/ciaox_install/Linux.190930/ciao-4.12/lib/python3.5/site-packages/pycrates/cratedata.py", line 100, in _set_values
    elif "array" in str(type(invals)) and (invals is not None and len(invals) > 0):
TypeError: len() of unsized object

Edited to add this looks like it is happening with a "scalar" value like the exposure time, since a simple pdb rub gives

TypeError: len() of unsized object
> /proj/xena/ciaox_install/Linux.190930/ciao-4.12/lib/python3.5/site-packages/pycrates/cratedata.py(100)_set_values()
     98 
     99             # table values already in an array
--> 100             elif "array" in str(type(invals)) and (invals is not None and len(invals) > 0):
    101                 # for single row data for all datatypes except bit/byte
    102                 if invals.size == 1 and "uint8" not in str(invals.dtype.type):

ipdb> invals                                                                    
array(43933.41642561)
ipdb> len(invals)                                                               
*** TypeError: len() of unsized object
ipdb> type(invals)                                                              
<class 'numpy.ndarray'>
ipdb> invals.size                                                               
1
ipdb> self.name                                                                 
'EXPOSURE'

[DougBurke]

So this appears to be specific to PHA data and occurs for both crates and astropy backends, but the error is different:

Crates

>>> from sherpa.astro.ui import *
>>> load_arrays(1, [1, 2, 3], [4, 2, -1])
>>> save_table('x', ascii=True, clobber=True)
>>> save_table('x', ascii=False, clobber=True)
>>> load_arrays(1, [1, 2, 3], [4, 2, 1], DataPHA)
>>> save_table('x', ascii=True, clobber=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in save_table
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/sherpa/astro/ui/utils.py", line 5027, in save_table
    sherpa.astro.io.write_table(filename, self.get_data(id),
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/sherpa/astro/io/__init__.py", line 736, in write_table
    backend.set_table_data(filename, data, names, ascii=ascii, clobber=clobber)
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/sherpa/astro/io/crates_backend.py", line 1250, in set_table_data
    _set_column(tbl, name, data[name])
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/sherpa/astro/io/crates_backend.py", line 190, in _set_column
    col.values = numpy.array(val)
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/pycrates/cratedata.py", line 100, in _set_values
    elif "array" in str(type(invals)) and (invals is not None and len(invals) > 0):
TypeError: len() of unsized object
>>> save_table('x', ascii=False, clobber=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in save_table
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/sherpa/astro/ui/utils.py", line 5027, in save_table
    sherpa.astro.io.write_table(filename, self.get_data(id),
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/sherpa/astro/io/__init__.py", line 736, in write_table
    backend.set_table_data(filename, data, names, ascii=ascii, clobber=clobber)
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/sherpa/astro/io/crates_backend.py", line 1250, in set_table_data
    _set_column(tbl, name, data[name])
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/sherpa/astro/io/crates_backend.py", line 190, in _set_column
    col.values = numpy.array(val)
  File "/home/dburke/anaconda/envs/ciao414b1/lib/python3.8/site-packages/pycrates/cratedata.py", line 100, in _set_values
    elif "array" in str(type(invals)) and (invals is not None and len(invals) > 0):
TypeError: len() of unsized object

AstroPy

>>> from sherpa.astro.ui import *
WARNING: imaging routines will not be available,
failed to import sherpa.image.ds9_backend due to
'RuntimeErr: DS9Win unusable: Could not find xpaget on your PATH'
WARNING: failed to import sherpa.astro.xspec; XSPEC models will not be available
>>> load_arrays(1, [1, 2, 3], [4, 2, -1])
>>> save_table('x', ascii=True, clobber=True)
>>> save_table('x', ascii=False, clobber=True)
>>> load_arrays(1, [1, 2, 3], [4, 2, 1], DataPHA)
>>> save_table('x', ascii=True, clobber=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in save_table
  File "/home/dburke/sherpa/sherpa-master/sherpa/astro/ui/utils.py", line 5027, in save_table
    --------
  File "/home/dburke/sherpa/sherpa-master/sherpa/astro/io/__init__.py", line 736, in write_table
    data, names = _pack_table(dataset)
  File "/home/dburke/sherpa/sherpa-master/sherpa/astro/io/pyfits_backend.py", line 1020, in set_table_data
    collist, cols, coldefs = _create_columns(col_names, data)
  File "/home/dburke/sherpa/sherpa-master/sherpa/astro/io/pyfits_backend.py", line 998, in _create_columns
    format=coldata.dtype.name.upper())
AttributeError: 'bool' object has no attribute 'dtype'
>>> save_table('x', ascii=False, clobber=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 1, in save_table
  File "/home/dburke/sherpa/sherpa-master/sherpa/astro/ui/utils.py", line 5027, in save_table
    --------
  File "/home/dburke/sherpa/sherpa-master/sherpa/astro/io/__init__.py", line 736, in write_table
    data, names = _pack_table(dataset)
  File "/home/dburke/sherpa/sherpa-master/sherpa/astro/io/pyfits_backend.py", line 1020, in set_table_data
    collist, cols, coldefs = _create_columns(col_names, data)
  File "/home/dburke/sherpa/sherpa-master/sherpa/astro/io/pyfits_backend.py", line 998, in _create_columns
    format=coldata.dtype.name.upper())
AttributeError: 'bool' object has no attribute 'dtype'