Catching odd object attributes

[din14970]

I'm trying to write out a hyperspy signal to USID, which uses sidpy under the hood. The signal was loaded from an experimental Velox EMD file. The full exception stack:

TypeError                                 Traceback (most recent call last)
<ipython-input-4-9b73ed519d2d> in <module>
----> 1 img[2].save("example.h5")

~/Documents/PythonProjects/FORKS/hyperspy/hyperspy/signal.py in save(self, filename, overwrite, extension, **kwds)
   2746         if extension is not None:
   2747             filename = filename.with_suffix(f".{extension}")
-> 2748         io.save(filename, self, overwrite=overwrite, **kwds)
   2749 
   2750     def _replot(self):

~/Documents/PythonProjects/FORKS/hyperspy/hyperspy/io.py in save(filename, signal, overwrite, **kwds)
    754         # Pass as a string for now, pathlib.Path not
    755         # properly supported in io_plugins
--> 756         writer.file_writer(str(filename), signal, **kwds)
    757 
    758         _logger.info(f'{filename} was created')

~/Documents/PythonProjects/FORKS/hyperspy/hyperspy/io_plugins/usid_hdf5.py in file_writer(filename, object2save, **kwds)
    505     if not append:
    506         tran = usid.NumpyTranslator()
--> 507         _ = tran.translate(filename, dset_name, data_2d, phy_quant, phy_units,
    508                            pos_dims, spec_dims, parm_dict=parm_dict,
    509                            slow_to_fast=True, **kwds)

~/opt/anaconda3/envs/hyperspy-dev/lib/python3.8/site-packages/pyUSID/io/numpy_translator.py in translate(self, h5_path, data_name, raw_data, quantity, units, pos_dims, spec_dims, translator_name, parm_dict, extra_dsets, **kwargs)
    126             # measurement group next
    127             meas_grp = create_indexed_group(h5_f, 'Measurement')
--> 128             write_simple_attrs(meas_grp, parm_dict)
    129 
    130             # channel group next

~/opt/anaconda3/envs/hyperspy-dev/lib/python3.8/site-packages/sidpy/hdf/hdf_utils.py in write_simple_attrs(h5_obj, attrs, verbose)
    400         if verbose:
    401             print('Attribute cleaned into: {}'.format(clean_val))
--> 402         h5_obj.attrs[key] = clean_val
    403     if verbose:
    404         print('Wrote all (simple) attributes to {}: {}\n'

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

~/opt/anaconda3/envs/hyperspy-dev/lib/python3.8/site-packages/h5py/_hl/attrs.py in __setitem__(self, name, value)
     98         use the methods create() and modify().
     99         """
--> 100         self.create(name, data=value)
    101 
    102     @with_phil

~/opt/anaconda3/envs/hyperspy-dev/lib/python3.8/site-packages/h5py/_hl/attrs.py in create(self, name, data, shape, dtype)
    182             # Make HDF5 datatype and dataspace for the H5A calls
    183             if use_htype is None:
--> 184                 htype = h5t.py_create(original_dtype, logical=True)
    185                 htype2 = h5t.py_create(original_dtype)  # Must be bit-for-bit representation rather than logical
    186             else:

h5py/h5t.pyx in h5py.h5t.py_create()

h5py/h5t.pyx in h5py.h5t.py_create()

h5py/h5t.pyx in h5py.h5t.py_create()

TypeError: Object dtype dtype('O') has no native HDF5 equivalent

[din14970]

Actually went in to check which value is giving the issue, it seems to be some odd part of the original metadata Writing attribute: Original-Operations-DisplayLevelsOperation-01661259da254a7190c741cfe208c723-outputs with value: [{'outputIndex': '0', 'operation': '/Presentation/Displays/ImageDisplay/2952d1eacfec4233a75934295d8d495c', 'inputIndex': '0'}]

[din14970]

for now I've solved it with:

try:
    h5_obj.attrs[key] = clean_val
except Exception:
    h5_obj.attrs[key] = str(clean_val)

But I guess that goes a bit against the philosophy of the way metadata should be stored.

[ssomnath]

@din14970 - Thank you for bringing this to our attention. I have implemented more safeguards against such situations. See this commit.

[ssomnath]

Anyone interested in improving testing for write_simple_attrs is welcome.

[din14970]

Thanks! I guess it's good to have safeguards in here as you implemented. A full fix for my case will probably require a look at how the hyperspy original metadata is mapped to tags in USID; I would guess it's a JSON flattening issue there. Will see whether I can find a solution there.