Databroker can't read Eiger images

[yugangzhang]

We can't read Eiger images under 'Collection'

In [65]: hdr = db[-1]

In [66]: hdr
Out[66]: 
{'descriptors': [{'data_keys': {'eiger_4M_cam_img_acquire_period': {'dtype': 'number',
     'shape': [],
     'source': 'PV:XF:11IDB-BI{Det:Eig4M}cam1:AcquirePeriod_RBV'},
    'eiger_4M_cam_img_acquire_time': {'dtype': 'number',
     'shape': [],
     'source': 'PV:XF:11IDB-BI{Det:Eig4M}cam1:AcquireTime_RBV'},
    'eiger_4M_cam_img_image_lightfield': {'dtype': 'array',
     'external': 'FILESTORE:',
     'shape': [2167, 2070],
     'source': 'PV:XF:11IDB-BI{Det:Eig4M}'},
    'eiger_4M_cam_img_stats_total1': {'dtype': 'number',
     'shape': [],
     'source': 'PV:XF:11IDB-BI{Det:Eig4M}Stats1:Total_RBV'},
    'eiger_4M_cam_img_stats_total2': {'dtype': 'number',
     'shape': [],
     'source': 'PV:XF:11IDB-BI{Det:Eig4M}Stats2:Total_RBV'},
    'eiger_4M_cam_img_stats_total3': {'dtype': 'number',
     'shape': [],
     'source': 'PV:XF:11IDB-BI{Det:Eig4M}Stats3:Total_RBV'},
    'eiger_4M_cam_img_stats_total4': {'dtype': 'number',
     'shape': [],
     'source': 'PV:XF:11IDB-BI{Det:Eig4M}Stats4:Total_RBV'},
    'eiger_4M_cam_img_stats_total5': {'dtype': 'number',
     'shape': [],
     'source': 'PV:XF:11IDB-BI{Det:Eig4M}Stats5:Total_RBV'},
    'eiger_4M_cam_imgsequenceid': {'dtype': 'number',
     'shape': [],
     'source': 'PV:XF:11IDB-BI{Det:Eig4M}cam1:SequenceId'}},
   'run_start': '574bb68b-f493-4f45-bd6e-fabc66c6f894',
   'time': 1444834144.1088412,
   'uid': '4ae706a4-3355-4b30-a866-c8779b3e9d9a'}],
 'start': {'beamline_id': 'CHX',
  'config': {},
  'group': 'chx',
  'owner': 'xf11id',
  'project': 'Undulator spectrum studies',
  'sample': {},
  'scan_args': {'delay': '0',
   'detectors': "[AreaDetectorFileStoreEiger(name='eiger_4M_cam_img', basename='XF:11IDB-BI{Det:Eig4M}', stats=range(1, 6), shutter=None, shutter_rb=None, shutter_val=None, file_path='/XF11ID/data/', ioc_file_path=None)]",
   'num': '1'},
  'scan_id': 289,
  'scan_type': 'Count',
  'time': 1444834132.0575428,
  'uid': '574bb68b-f493-4f45-bd6e-fabc66c6f894'},
 'stop': {'exit_status': 'success',
  'reason': '',
  'run_start': '574bb68b-f493-4f45-bd6e-fabc66c6f894',
  'time': 1444834144.1293712,
  'uid': 'e39e559e-57e8-4883-9fb4-adb8d7742596'}}

In [67]: img = get_images( hdr, 'eiger_4M_cam_img_image_lightfield' )
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-67-a307ab930b8b> in <module>()
----> 1 img = get_images( hdr, 'eiger_4M_cam_img_image_lightfield' )

/opt/conda_envs/collection/lib/python3.4/site-packages/databroker-0.3.0-py3.4.egg/databroker/pims_readers.py in get_images(headers, name)
     23             # do something
     24     """
---> 25     return Images(headers, name)
     26 
     27 

/opt/conda_envs/collection/lib/python3.4/site-packages/databroker-0.3.0-py3.4.egg/databroker/pims_readers.py in __init__(self, headers, name)
     47         self._datum_uids = [event.data[name] for event in events]
     48         self._len = len(self._datum_uids)
---> 49         example_frame = retrieve(self._datum_uids[0])
     50         self._dtype = example_frame.dtype
     51         self._shape = example_frame.shape

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/commands.py in inner(*args, **kwargs)
     19         port = int(conf.connection_config['port'])
     20         db_connect(database=database, host=host, port=port)
---> 21         return func(*args, **kwargs)
     22     return inner
     23 

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/commands.py in retrieve(eid)
    136         The requested data as a numpy array
    137     """
--> 138     return _get_data(eid)

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/retrieve.py in get_data(eid, handle_registry)
    261                         "datum cache can hold.")
    262 
--> 263     handler = get_spec_handler(datum['resource'], handle_registry)
    264     return handler(**datum['datum_kwargs'])

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/retrieve.py in get_spec_handler(resource, handle_registry)
    204 
    205     spec = resource['spec']
--> 206     handler = handle_registry[spec]
    207     key = (str(resource['_id']), handler.__name__)
    208     if key in _HANDLER_CACHE:

KeyError: 'AD_EIGER'

We also don't have EigerImages functions in chxtools.pims_reader

In [69]: from chxtools.pims_readers import EigerImages
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-69-330a75e1c6f7> in <module>()
----> 1 from chxtools.pims_readers import EigerImages

/home/xf11id/shared/chxtools/chxtools/pims_readers/__init__.py in <module>()
----> 1 from eiger import EigerImages
      2 from edf import EDFImages

ImportError: cannot import name 'EigerImages'

In [70]: 

What happed?

[tacaswell]

Because you have not registered the eiger handler. Someplace in your other code there is something like register_handler('AD_EIGER', ....). You just need to run the same code in your collection environment.

[yugangzhang]

But, I don't have access to modify the /opt/... code. How can run it under collection, using the commandline?

In [70]: register_handler
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-70-7ebeee835877> in <module>()
----> 1 register_handler

NameError: name 'register_handler' is not defined

[tacaswell]

See http://nsls-ii.github.io/filestore/#the-payoff-retrieving-data-is-dead-simple

[ericdill]

try from chxtools.pims_readers.eiger import EigerImages

[ericdill]

Ah, the problem is the bad import line in chxtools.pims_readers.init.py. I'll put in a PR to fix this

[ericdill]

https://github.com/NSLS-II-CHX/chxtools/pull/5

[ericdill]

@yugangzhang have a look at this PR and merge it if you are :+1:

[yugangzhang]

I wrote a plot_sid function in chxtools/chxtools/ yesterday and will put to github.

[yugangzhang]

@tacaswell That's what I did for registered the eiger handler:

In [83]: from filestore.api import register_handler, retrieve

In [84]: class HDF5DatasetHandler(object):
       ....:     def __init__(self, filename):
           ....:         self.file = h5py.File(filename)
           ....:     def __call__(self, key):
           ....:         return self.file[key].value
   ....:     
In [85]: register_handler('AD_EIGER',  HDF5DatasetHandler)

But I got a wrong message:

In [86]: img = get_images( hdr, 'eiger_4M_cam_img_image_lightfield' )
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-86-a307ab930b8b> in <module>()
----> 1 img = get_images( hdr, 'eiger_4M_cam_img_image_lightfield' )

/opt/conda_envs/collection/lib/python3.4/site-packages/databroker-0.3.0-py3.4.egg/databroker/pims_readers.py in get_images(headers, name)
     23             # do something
     24     """
---> 25     return Images(headers, name)
     26 
     27 

/opt/conda_envs/collection/lib/python3.4/site-packages/databroker-0.3.0-py3.4.egg/databroker/pims_readers.py in __init__(self, headers, name)
     47         self._datum_uids = [event.data[name] for event in events]
     48         self._len = len(self._datum_uids)
---> 49         example_frame = retrieve(self._datum_uids[0])
     50         self._dtype = example_frame.dtype
     51         self._shape = example_frame.shape

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/commands.py in inner(*args, **kwargs)
     19         port = int(conf.connection_config['port'])
     20         db_connect(database=database, host=host, port=port)
---> 21         return func(*args, **kwargs)
     22     return inner
     23 

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/commands.py in retrieve(eid)
    136         The requested data as a numpy array
    137     """
--> 138     return _get_data(eid)

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/retrieve.py in get_data(eid, handle_registry)
    261                         "datum cache can hold.")
    262 
--> 263     handler = get_spec_handler(datum['resource'], handle_registry)
    264     return handler(**datum['datum_kwargs'])

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/retrieve.py in get_spec_handler(resource, handle_registry)
    210     kwargs = resource['resource_kwargs']
    211     rpath = resource['resource_path']
--> 212     ret = handler(rpath, **kwargs)
    213     _HANDLER_CACHE[key] = ret
    214     return ret

TypeError: __init__() got an unexpected keyword argument 'frame_per_point'

[danielballan]

You didn't register the EigerHandler. You registered the HDF5DatasetHandler, which correctly raised because it does not know how to handle Eiger's HDF5 files.

[tacaswell]

Also, don't use self.file[key].value, use self.file[key][:] instead.

[yugangzhang]

@tacaswell and @danielballan , I change the code:

In [4]: import h5py
In [5]: class EigerHandler(object):
    def __init__(self, filename):
        self.file = h5py.File(filename)
        def __call__(self, key):
            return self.file[key][:]
   ...:         

In [6]: from filestore.api import register_handler, retrieve

In [7]: register_handler('AD_EIGER', EigerHandler)

In [8]: hdr=db[-1]

I still got a same wrong message:

In [9]: img = get_images( hdr, 'eiger_4M_cam_img_image_lightfield' )
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-9-a307ab930b8b> in <module>()
----> 1 img = get_images( hdr, 'eiger_4M_cam_img_image_lightfield' )

/opt/conda_envs/collection/lib/python3.4/site-packages/databroker-0.3.0-py3.4.egg/databroker/pims_readers.py in get_images(headers, name)
     23             # do something
     24     """
---> 25     return Images(headers, name)
     26 
     27 

/opt/conda_envs/collection/lib/python3.4/site-packages/databroker-0.3.0-py3.4.egg/databroker/pims_readers.py in __init__(self, headers, name)
     47         self._datum_uids = [event.data[name] for event in events]
     48         self._len = len(self._datum_uids)
---> 49         example_frame = retrieve(self._datum_uids[0])
     50         self._dtype = example_frame.dtype
     51         self._shape = example_frame.shape

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/commands.py in inner(*args, **kwargs)
     19         port = int(conf.connection_config['port'])
     20         db_connect(database=database, host=host, port=port)
---> 21         return func(*args, **kwargs)
     22     return inner
     23 

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/commands.py in retrieve(eid)
    136         The requested data as a numpy array
    137     """
--> 138     return _get_data(eid)

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/retrieve.py in get_data(eid, handle_registry)
    261                         "datum cache can hold.")
    262 
--> 263     handler = get_spec_handler(datum['resource'], handle_registry)
    264     return handler(**datum['datum_kwargs'])

/opt/conda_envs/collection/lib/python3.4/site-packages/filestore/retrieve.py in get_spec_handler(resource, handle_registry)
    210     kwargs = resource['resource_kwargs']
    211     rpath = resource['resource_path']
--> 212     ret = handler(rpath, **kwargs)
    213     _HANDLER_CACHE[key] = ret
    214     return ret

TypeError: __init__() got an unexpected keyword argument 'frame_per_point'

[tacaswell]

All you did was change the class name, the body is still the same.

The eiger saves many frames per point, the frame_per_point kwarg tells the handler how many frames to extract from the hdf5 file per 'datum'.

[yugangzhang]

No, actually I changed from return self.file.value to return self.file[key][:] I guess need to do more. I can't use my computer to debug due to #37, @tacaswell do you have a suggestion? Now, I have to use the beamline computer to do debug, painful!!!

[danielballan]

Tom confused the issue; his comment about [:] was an unrelated issue.

You need to use the actual Eiger...Handler class which is not the same as the HDF5...Handler we wrote for CSX's detector. That class is, I believe, defined in chxtools.

[danielballan]

Can you debug through a notebook using jupyterhub?

[yugangzhang]

Great! The jupyterhub works! I will debug by it.

[yugangzhang]

@danielballan I looked through chxtools and only found a class

class EigerImages(FramesSequence):

and did not find the definition of

 Eiger...Handler 

[yugangzhang]

@danielballan on jupyterhub, we don't have chxtools

import chxtools
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-5-886d7eb7d21e> in <module>()
----> 1 import chxtools

ImportError: No module named 'chxtools'

I also can't install chxtools in jupyterhub, suggestion?

[danielballan]

Just to make sure we are clear, the ... stands for "I don't remember the exactly name."

Re, chxtools in the jupyterhub environment: Ah, I should have done that for you. This will have to wait for tomorrow now.

Meanwhile, we need to find this missing handler code. Tom and I wrote it and ran for the train. I remember cleaning it up somewhat recently, but I can't remember where we put it. It doesn't seem to be in chxtools on github. Could it be in a local branch? I know we at least committed it. Does chxtools need to be pushed to github?

[yugangzhang]

@danielballan, I know your meaning for ... :) For Jupterhub, can we have access to install code? I will try to find the missing handler code. We have a chxtools github https://github.com/NSLS-II-CHX/chxtools

[danielballan]

Yeah I know you have it on GitHub, but I'm questioning whether GitHub's copy is up to date with your local copy.

Customizable jupyterhub environments are coming. I have a way to do it but no time yet to set it up. On Wed, Oct 14, 2015 at 5:47 PM Yugang Zhang notifications@github.com wrote:

@danielballan https://github.com/danielballan, I know your meaning for python ... :) For Jupterhub, can we have access to install code? I will try to find the missing handler code. We have a chxtools github https://github.com/NSLS-II-CHX/chxtools

— Reply to this email directly or view it on GitHub https://github.com/NSLS-II/Bug-Reports/issues/36#issuecomment-148211890.

[yugangzhang]

Yes, I check our localone and the git one. They are same now. I guess Eirc just update them. So, I need to check other place for the missing code. Thanks for the jupter customize!

[danielballan]

It's conceivable that the latest version of that code fell into the window of what was lost during the NFS failure, but at minimum some version of this code has been around for awhile. It must be there somewhere. On Wed, Oct 14, 2015 at 5:55 PM Yugang Zhang notifications@github.com wrote:

Yes, I check our localone and the git one. They are same now. I guess Eirc just update them.

— Reply to this email directly or view it on GitHub https://github.com/NSLS-II/Bug-Reports/issues/36#issuecomment-148213354.

[yugangzhang]

Up to now, I did not find it. We might need you come to our beamline tomorrow. ...

[yugangzhang]

I still did not mange to open eiger by databroker, then I tried to open a disk file. I found a problem of HDF5_PLUGIN_PATH.

In [21]: img = entry[ keys[0] ][0]
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-21-98d831a2dbe5> in <module>()
----> 1 img = entry[ keys[0] ][0]

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2458)()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2415)()

/nfs/xf11id/mc/envs/ophyd/lib/python2.7/site-packages/h5py/_hl/dataset.pyc in __getitem__(self, args)
    449         mspace = h5s.create_simple(mshape)
    450         fspace = selection._id
--> 451         self.id.read(mspace, fspace, arr, mtype)
    452 
    453         # Patch up the output for NumPy

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2458)()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2415)()

h5py/h5d.pyx in h5py.h5d.DatasetID.read (-------src-dir-------/h5py/h5d.c:3058)()

h5py/_proxy.pyx in h5py._proxy.dset_rw (-------src-dir-------/h5py/_proxy.c:1670)()

h5py/_proxy.pyx in h5py._proxy.H5PY_H5Dread (-------src-dir-------/h5py/_proxy.c:1336)()

IOError: Can't read data (Required filter 'hdf5 lz4 filter; see http://www.hdfgroup.org/services/contributions.html' is not registered)

I knew this problem came from a hdf5-plugin path.

On our previous ophyd, we use
python export HDF5_PLUGIN_PATH=/nfs/xf11id/mc/envs/ophyd/lib and we can open a eiger-hdf5 file.

Now, on current collection, I tried

python export HDF5_PLUGIN_PATH=/nfs/xf11id/mc/envs/ophyd/lib but, the collection doesn't work. I guess we have to use /opt/ path so I changed it to

python export HDF5_PLUGIN_PATH=/opt/conda_envs/collection/lib But, I got a same problem.

In [21]: img = entry[ keys[0] ][0]
---------------------------------------------------------------------------
IOError                                   Traceback (most recent call last)
<ipython-input-21-98d831a2dbe5> in <module>()
----> 1 img = entry[ keys[0] ][0]

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2458)()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2415)()

/nfs/xf11id/mc/envs/ophyd/lib/python2.7/site-packages/h5py/_hl/dataset.pyc in __getitem__(self, args)
    449         mspace = h5s.create_simple(mshape)
    450         fspace = selection._id
--> 451         self.id.read(mspace, fspace, arr, mtype)
    452 
    453         # Patch up the output for NumPy

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2458)()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper (-------src-dir-------/h5py/_objects.c:2415)()

h5py/h5d.pyx in h5py.h5d.DatasetID.read (-------src-dir-------/h5py/h5d.c:3058)()

h5py/_proxy.pyx in h5py._proxy.dset_rw (-------src-dir-------/h5py/_proxy.c:1670)()

h5py/_proxy.pyx in h5py._proxy.H5PY_H5Dread (-------src-dir-------/h5py/_proxy.c:1336)()

IOError: Can't read data (Required filter 'hdf5 lz4 filter; see http://www.hdfgroup.org/services/contributions.html' is not registered)

[yugangzhang]

Another problem for a piece of code in /home/xf11id/shared/chxtools/chxtools/pims_readers/eiger.py

under ophyd:

In [13]: lengths
Out[13]: [1]
In [14]: toc = np.concatenate(
   ....:                 [zip(i*np.ones(length, dtype=int),
   ....:                      np.arange(length, dtype=int))
   ....:                 for i, length in enumerate(lengths)])

In [15]: toc
Out[15]: array([[0, 0]])

but, using collection,

In [13]: lengths
Out[13]: [1]

In [14]: toc = np.concatenate(
   ....:                 [zip(i*np.ones(length, dtype=int),
   ....:                      np.arange(length, dtype=int))
   ....:                 for i, length in enumerate(lengths)])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-14-a479f3fe7f3b> in <module>()
      2                 [zip(i*np.ones(length, dtype=int),
      3                      np.arange(length, dtype=int))
----> 4                 for i, length in enumerate(lengths)])

ValueError: zero-dimensional arrays cannot be concatenated

I guess a python version problem?

[yugangzhang]

(1) chxtools.pims_readers.eiger.py can read eiger hdf5 file now. I found a bug in chxtools.pims_readers.eiger.py, the self._toc works well for python 2, but not for python 3. I modified self._toc and the eiger.py can read a eiger file now:

In [8]: from chxtools.pims_readers.eiger import EigerImages as Images
In [9]: path ='/XF11ID/data/2015/10/14/'
In [10]: fn = 'fe294a59-f2e4-477a-8e88_15_master.h5'
In [11]: fp = path + fn
In [12]: img = Images( fp)
In [13]: img
Out[13]: 
<Frames>
Length: 1 frames
Frame Shape: 2167 x 2070
Pixel Datatype: uint32

(2) a problem for Handler class. I did not find @danielballan and @tacaswell 's python Eiger...Handler class, then I wrote a class:

from chxtools.pims_readers.eiger import EigerImages
class HDF5Handler(object):   
   def __init__(self, filename):
      self.filename =  filename 
   def __call__(self):
      return  EigerImages( self.filename )

to register 'AD_EIGER'.

But, when I try to read filestore data, i got a wrong message "unexpected keyword argument 'frame_per_point'" as previously.

In [68]: from databroker import DataBroker as db, get_events, get_images, get_table

In [69]: hdr = db['fedb7385-3255-43fd-9785']

In [70]: name = 'eiger_4M_cam_img_image_lightfield'

In [71]: get_images( hdr,name )
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-71-ddd2768ec494> in <module>()
----> 1 get_images( hdr,name )

/opt/conda_envs/analysis/lib/python3.4/site-packages/databroker-0.3.0-py3.4.egg/databroker/pims_readers.py in get_images(headers, name)
     23             # do something
     24     """
---> 25     return Images(headers, name)
     26 
     27 

/opt/conda_envs/analysis/lib/python3.4/site-packages/databroker-0.3.0-py3.4.egg/databroker/pims_readers.py in __init__(self, headers, name)
     47         self._datum_uids = [event.data[name] for event in events]
     48         self._len = len(self._datum_uids)
---> 49         example_frame = retrieve(self._datum_uids[0])
     50         self._dtype = example_frame.dtype
     51         self._shape = example_frame.shape

/opt/conda_envs/analysis/lib/python3.4/site-packages/filestore/commands.py in inner(*args, **kwargs)
     19         port = int(conf.connection_config['port'])
     20         db_connect(database=database, host=host, port=port)
---> 21         return func(*args, **kwargs)
     22     return inner
     23 

/opt/conda_envs/analysis/lib/python3.4/site-packages/filestore/commands.py in retrieve(eid)
    136         The requested data as a numpy array
    137     """
--> 138     return _get_data(eid)

/opt/conda_envs/analysis/lib/python3.4/site-packages/filestore/retrieve.py in get_data(eid, handle_registry)
    261                         "datum cache can hold.")
    262 
--> 263     handler = get_spec_handler(datum['resource'], handle_registry)
    264     return handler(**datum['datum_kwargs'])

/opt/conda_envs/analysis/lib/python3.4/site-packages/filestore/retrieve.py in get_spec_handler(resource, handle_registry)
    210     kwargs = resource['resource_kwargs']
    211     rpath = resource['resource_path']
--> 212     ret = handler(rpath, **kwargs)
    213     _HANDLER_CACHE[key] = ret
    214     return ret

TypeError: __init__() got an unexpected keyword argument 'frame_per_point'

I wonder should I add the 'frame_per_point' keyword to my handler class? Or there is a bug in filestore.api.retrieve fucntion?

[danielballan]

Yes, you should add frame_per_point keyword to your handler class. In our Eiger...Handler class there was quite a bit of logic to handle the particular structure of the Eiger's HDF5 files. You will need to replicate that as well.

[yugangzhang]

Aha! Do you have any clue to find it?

[ericdill]

With these commands, you can find the EigerHandler:

$ cd /home/xf11id/Repos
$ grep -r Eiger
...
1de40 Daron Chabot <dchabot@bnl.gov> 1438798631 -0400 commit: ENH: Add
EigerHandler.
filestore/.git/COMMIT_EDITMSG:ENH: Add EigerHandler.
filestore/filestore/handlers.py:from chxtools.pims_readers import
EigerImages
filestore/filestore/handlers.py:class EigerHandler(HandlerBase):
filestore/filestore/handlers.py:        return
np.array(EigerImages(master_path))
Binary file filestore/filestore/handlers.pyc matches
$ cat filestore/filestore/handlers.py

class EigerHandler(HandlerBase):
    specs = {'AD_EIGER'} | HandlerBase.specs

    def __init__(self, fpath, frame_per_point):
        # create pims handler
        self._base_path = fpath
        self.fpp = frame_per_point

    def __call__(self, seq_id):
        master_path = '{}_{}_master.h5'.format(self._base_path, seq_id)
        # TODO Return a multi-dimensional PIMS seq.
        return np.array(EigerImages(master_path))

I would imagine that is the handler that you are looking for.

On Fri, Oct 16, 2015 at 11:39 AM, Yugang Zhang notifications@github.com wrote:

Aha! Do you have any clue to find it?

— Reply to this email directly or view it on GitHub https://github.com/NSLS-II/Bug-Reports/issues/36#issuecomment-148749573.

Dr. Eric Dill Assistant Computational Scientist National Synchrotron Light Source 2

[danielballan]

@ericdill Wait, so that code is in a rogue branch of filestore that has a dependency on chxtools?

[ericdill]

Sure looks that way

On Fri, Oct 16, 2015, 11:59 AM Dan Allan notifications@github.com wrote:

@ericdill https://github.com/ericdill Wait, so that code is in a rogue branch of filestore that has a dependency on chxtools?

— Reply to this email directly or view it on GitHub https://github.com/NSLS-II/Bug-Reports/issues/36#issuecomment-148754807.

[danielballan]

@yugangzhang Please copy the code for that class into chxtools. Let me know when it is done and please paste that path to chxtools into this issue. Then I will install chxtools into collection and the jupyterhub env.

As we've discussed, I do plan to make it possible for you to do this yourself, but I can't take time to set that up just now.

[danielballan]

@yugangzhang I changed my mind. I will make a special (very small) package just for that handler. I've done this kind of thing for pims, and I think it makes things simple. You don't have to do anything. Give me a couple minutes....

[yugangzhang]

Thanks to @ericdill , it works now!

In [9]: hdr = db['fedb7385-3255-43fd-9785']

In [10]: name = 'eiger_4M_cam_img_image_lightfield'

In [11]: imgs = get_images( hdr,name )

In [12]: imgs
Out[12]: 
<Frames>
Length: 1 frames
Frame Shape: 1 x 2167
Pixel Datatype: uint32

It's great! In the future, I would like to change the Frame Shape: 1 x 2167 to 1X2167X...

[danielballan]

@yugangzhang How did you get that working? In what environment?

[yugangzhang]

@danielballan , I just did... It might not be the best way. I did it in analysis environment.

[danielballan]

My guess: you happened to start ipython while sitting in the Repos directory, so the "rogue" branch of filestore on your system overrode the version installed in the analysis environment.

[yugangzhang]

Half True. I wrote a new hander.py and load it under analysis.

[yugangzhang]

Talk more after lunch.

[danielballan]

Taking a step back...

It's generally good and useful to put separate concerns like analysis and I/O in separate repos. They have very different dependencies and usually different "rates of change", meaning frequency with which they are edited/updated.

To that end, I created a new project, eiger-io, uploaded it to github, and installed it in collection and analysis on xf11id-ws1.

In [5]: h = db['574bb68b']

In [6]: from filestore.api import register_handler

In [7]: import eiger_io.fs_handler

In [8]: register_handler('AD_EIGER', eiger_io.fs_handler.EigerHandler)

In [9]: get_images(h, 'eiger_4M_cam_img_image_lightfield')

For me, that reproduces the error you reported above:

ValueError: zero-dimensional arrays cannot be concatenated

That's what we will address next.

[danielballan]

cc @klauer @ericdill Can I call in some help debugging it from here?

[tacaswell]

Sorry for making this mess!

On Fri, Oct 16, 2015, 13:03 Dan Allan notifications@github.com wrote:

Taking a step back...

It's generally good and useful to put separate concerns in separate repos. They have very different dependencies and usually different "rates of change", meaning frequency with which they are edited/updated.

To that end, I created a new project, eiger-io, uploaded it to github https://github.com/NSLS-II-CHX/eiger-io, and installed it in collection and analysis on xf11id-ws1.

In [6]: from filestore.api import register_handler

In [7]: import eiger_io

In [8]: register_handler('AD_EIGER', eiger_io.fs_handler.EigerHandler)

In [9]: get_images(h, 'eiger_4M_cam_img_image_lightfield')

For me, that reproduces the error you reported above:

ValueError: zero-dimensional arrays cannot be concatenated

That's what we will address next.

— Reply to this email directly or view it on GitHub https://github.com/NSLS-II/Bug-Reports/issues/36#issuecomment-148773028.

[yugangzhang]

@danielballan , python ValueError: zero-dimensional arrays cannot be concatenated That's the bug I talked about in code "self._toc" of chxtools.pims_readers.eiger.py, the self._toc works well for python 2, but not for python 3. I already modified self._toc.

[yugangzhang]

change

self._toc = np.concatenate(
 [zip(i*np.ones(length, dtype=int),
np.arange(length, dtype=int))
for i, length in enumerate(lengths)])

to

        toc=[]
        for i, length in enumerate(lengths):   
           a,b=i*np.ones(length, dtype=int), np.arange(length, dtype=int)
           #print (a,b)
           toc.append([ [x,y] for [x,y] in zip(a,b) ])
        self._toc = np.concatenate(toc )

[danielballan]

Why?

[danielballan]

Oh, I think I know why. Could you try this simpler change:

self._toc = np.concatenate(
 [list(zip(i*np.ones(length, dtype=int)),
np.arange(length, dtype=int))
for i, length in enumerate(lengths)])

If you think it works, please open a pull request against the eiger-io repo on GitHub. It will be good Github practice!

[danielballan]

To elaborate on my last comment, this is the specific change between python 2 and 3:

Python 3:

In [1]: zip([1,2,3], [4,5,6])  # lazily evaluated, not immediately turned into a list
Out[1]: <zip at 0x7f696daa2a08>

In [2]: list(zip([1,2,3], [4,5,6]))
Out[2]: [(1, 4), (2, 5), (3, 6)]

Python 2:

In [1]: zip([1,2,3], [4,5,6])  # returns a list
Out[1]: [(1, 4), (2, 5), (3, 6)]

[yugangzhang]

Ok! I assume it works and I will make a pull request.