Get_images can't open some files recorded by Eiger at CHX

[yugangzhang]

At CHX, we met a problem that the function get_image (/opt/conda_envs/analysis/lib/python3.5/site-packages/databroker/pims_readers.py) can't open some files recorded by Eiger4M detector. We knew that some frames (typically the first several frames) of that files were broken, but most of the frames are good and we can open them by albula, a GUI provided by Eiger company. Here is an example:

uid = '0da0ea'
imgs = get_images( db[uid ], name = 'eiger4m_single_image' )

and we got

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-194-a7fc6d014e88> in <module>()
----> 1 imgs = get_images( db[uid ], name = 'eiger4m_single_image' )

/opt/conda_envs/analysis/lib/python3.5/site-packages/databroker/pims_readers.py in get_images(headers, name, handler_registry, handler_override)
     29     res = DataBroker.get_images(headers=headers, name=name,
     30                                 handler_registry=handler_registry,
---> 31                                 handler_override=handler_override)
     32     return res
     33 

/opt/conda_envs/analysis/lib/python3.5/site-packages/databroker/broker.py in get_images(self, headers, name, handler_registry, handler_override)
    388         """
    389         return Images(self.mds, self.fs, headers, name, handler_registry,
--> 390                       handler_override)
    391 
    392 

/opt/conda_envs/analysis/lib/python3.5/site-packages/databroker/core.py in __init__(self, mds, fs, headers, name, handler_registry, handler_override)
    552         # Python object.
    553         try:
--> 554             self._dtype = example_frame.dtype
    555         except AttributeError:
    556             self._dtype = type(example_frame)

/opt/conda_envs/analysis/lib/python3.5/site-packages/eiger_io-0+untagged.31.g8c3bb99-py3.5.egg/eiger_io/fs_handler2.py in dtype(self)
     50     @property
     51     def dtype(self):
---> 52         return self.pixel_type
     53 
     54     @property

/opt/conda_envs/analysis/lib/python3.5/site-packages/eiger_io-0+untagged.31.g8c3bb99-py3.5.egg/eiger_io/fs_handler2.py in pixel_type(self)
     46     @property
     47     def pixel_type(self):
---> 48         return self[0].dtype
     49 
     50     @property

/opt/conda_envs/analysis/lib/python3.5/site-packages/slicerator-0.9.7-py3.5.egg/slicerator.py in __getitem__(self, i)
    192                 indices, new_length = key_to_indices(i, len(self))
    193                 if new_length is None:
--> 194                     return self._get(indices)
    195                 else:
    196                     return cls(self, indices, new_length, propagate_attrs)

/opt/conda_envs/analysis/lib/python3.5/site-packages/pims/base_frames.py in __getitem__(self, key)
    150         """__getitem__ is handled by Slicerator. In all pims readers, the data
    151         returning function is get_frame."""
--> 152         return self.get_frame(key)
    153 
    154     def __iter__(self):

/opt/conda_envs/analysis/lib/python3.5/site-packages/eiger_io-0+untagged.31.g8c3bb99-py3.5.egg/eiger_io/fs_handler2.py in get_frame(self, i)
     33 
     34     def get_frame(self, i):
---> 35         dataset = self._entry['data_{:06d}'.format(1 + (i // self.images_per_file))]
     36         img = dataset[i % self.images_per_file]
     37         return Frame(img, frame_no=i)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)()

/opt/conda_envs/analysis/lib/python3.5/site-packages/h5py/_hl/group.py in __getitem__(self, name)
    164                 raise ValueError("Invalid HDF5 object reference")
    165         else:
--> 166             oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
    167 
    168         otype = h5i.get_type(oid)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2696)()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (/home/ilan/minonda/conda-bld/work/h5py/_objects.c:2654)()

h5py/h5o.pyx in h5py.h5o.open (/home/ilan/minonda/conda-bld/work/h5py/h5o.c:3582)()

KeyError: "Unable to open object (Unable to open file: name = '/xf11id/data/2017/06/16/2f5aa51d-b537-4e61-87da_607_data_000001.h5', errno = 2, error message = 'no such file or directory', flags = 0, o_flags = 0)"

Can we add options, like 'good_start/stop_frame', in the get_images function? So, if the func can't open some frames, it give a warning message for that bad frames and continue to give all the good frames.

[tacaswell]

I suggest implementing this in the Eiger handler to return a frame of np.nan on this sort of failure.

[jrmlhermitte]

I don't see this error with this uid. Do you have another uid to test on?

uid = '0da0ea'
imgs = db['uid'].data('eiger4m_single_image')
for img in imgs:
    print(img[0])