search

APS-2BM-MIC / ipython-user2bmb

ipython configurations for the tomography instrument
2 stars 5 forks source link

MyHDF5Plugin() attribute returns numpy object, not str #13

Open prjemian opened 6 years ago

prjemian commented 6 years ago

Given an area detector constructed with this class:

class MyHDF5Plugin(HDF5Plugin, FileStoreHDF5IterativeWrite):
    """adapt HDF5 plugin for AD 2.5+"""

    file_number_sync = None
    xml_layout_file = Component(EpicsSignalWithRBV, "XMLFileName", string=True)
    xml_layout_valid = Component(EpicsSignalRO, "XMLValid_RBV")
    xml_layout_error_message = Component(EpicsSignalRO, "XMLErrorMsg_RBV", string=True)

    def get_frames_per_point(self):
        return self.parent.cam.num_images.get()

The xml_layout_file attribute returns a numpy object, rather than a string:

In [13]: pco_edge.hdf1.xml_layout_file.value
Out[13]: '<array size=21, type=time_char>'

In [14]: pco_edge.hdf1.xml_layout_file
Out[14]: EpicsSignalWithRBV(read_pv='PCOIOC3:HDF1:XMLFileName_RBV', name='pco_edge_hdf1_xml_layout_file', parent='pco_edge_hdf1', value='<array size=21, type=time_char>', timestamp=1518104101.386746, pv_kw={}, auto_monitor=False, string=True, write_pv='PCOIOC3:HDF1:XMLFileName', limits=False, put_complete=False)

The expected values are:

In [16]: !caget -S PCOIOC3:HDF1:XMLFileName_RBV PCOIOC3:HDF1:XMLFileName
PCOIOC3:HDF1:XMLFileName_RBV DynaMCTHDFLayout.xml
PCOIOC3:HDF1:XMLFileName DynaMCTHDFLayout.xml
prjemian commented 6 years ago

This is interesting but not critical for the beam time this weekend.

prjemian commented 6 years ago

Still a problem:

In [85]: !caget -S 2bmbPG3:HDF1:XMLFileName_RBV
2bmbPG3:HDF1:XMLFileName_RBV monaLayout.xml

In [86]: xmlFileName = EpicsSignalWithRBV("2bmbPG3:HDF1:XMLFileName", name="xmlFileName", string=True)

In [87]: xmlFileName.value
Out[87]: '<array size=15, type=time_char>'
prjemian commented 6 years ago

Test if this is solved by replacement with APS_BlueSky_tools.devices.ApsHDF5Plugin()

prjemian commented 6 years ago

can solve this with no beam, move to later milestone

prjemian commented 5 years ago 
In [5]: !caget -S 2bmbPG3:HDF1:XMLFileName
2bmbPG3:HDF1:XMLFileName monaLayout.xml

In [6]: !caget -S 2bmbPG3:HDF1:XMLFileName.RTYP
2bmbPG3:HDF1:XMLFileName.RTYP  waveform

In [7]: xmlFileName = EpicsSignalWithRBV("2bmbPG3:HDF1:XMLFileName", name="xmlFileName", string=True)

In [8]: xmlFileName.value
Out[8]: '<array size=15, type=time_char>'

In [9]: xmlFileName = EpicsSignal("2bmbPG3:HDF1:XMLFileName", name="xmlFileName", string=True)

In [10]: xmlFileName.value
Out[10]: '<array size=15, type=time_char>'

In [11]: xmlFileName = EpicsSignal("2bmbPG3:HDF1:XMLFileName", name="xmlFileName")

In [12]: xmlFileName.value
Out[12]: 
array([109, 111, 110,  97,  76,  97, 121, 111, 117, 116,  46, 120, 109,
       108,   0], dtype=uint8)
prjemian commented 5 years ago

Looks like a general problem for ophyd to resolve. @tacaswell , @danielballan : ideas? This is a field used in the Area Detector HDF5 File Writer plugin. It is a loooong string (waveform string). I expect it to be reported as a string, not a numpy object. Where to fix this?

prjemian commented 5 years ago

moving to next milestone since the fix is not essential for operations now

tacaswell commented 5 years ago

If you use .read() does it behave as expected?

prjemian commented 5 years ago

No, it behaves as reported:

[user2bmb@lyra,42,~]$ bash
user2bmb@lyra ~ $ source /APSshare/anaconda3/BlueSky/bin/activate base
(base) user2bmb@lyra ~ $ ipython
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from ophyd import EpicsSignal

In [2]: xmlFileName = EpicsSignalWithRBV("2bmbPG3:HDF1:XMLFileName", name="xmlFileName", string=True)
---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
<ipython-input-2-1e4467b787ca> in <module>()
----> 1 xmlFileName = EpicsSignalWithRBV("2bmbPG3:HDF1:XMLFileName", name="xmlFileName", string=True)

NameError: name 'EpicsSignalWithRBV' is not defined

In [3]: from ophyd import EpicsSignalWithRBV

In [4]: xmlFileName = EpicsSignalWithRBV("2bmbPG3:HDF1:XMLFileName", name="xmlFileName", string=True)

In [5]: xmlFileName.read()
Out[5]:
{'xmlFileName': {'value': '<array size=15, type=time_char>',
  'timestamp': 1543900458.677582}}

In [6]: !caget -S 2bmbPG3:HDF1:XMLFileName
2bmbPG3:HDF1:XMLFileName monaLayout.xml
prjemian commented 5 years ago

@newville : this may be a job for PyEpics to resolve:

In [13]: import epics

In [14]: pv = epics.PV("2bmbPG3:HDF1:XMLFileName")

In [15]: pv.get()
Out[15]:
array([109, 111, 110,  97,  76,  97, 121, 111, 117, 116,  46, 120, 109,
       108,   0], dtype=uint8)

In [16]: pv.get(as_string=True)
Out[16]: '<array size=15, type=time_char>'
newville commented 5 years ago

@prjemian Hm, I'm not sure. Maybe a pyepics version? I see this (with pyepics 3.3.1)

Python 3.6.6 |Anaconda custom (64-bit)| (default, Oct  9 2018, 12:34:16) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.2.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import epics                                                                                                                                     
In [2]: pv = epics.PV('13IDEPS1:TIFF1:FileName')                                                                                                         
In [3]: pv                                                                                                                                               
Out[3]: <PV '13IDEPS1:TIFF1:FileName', count=13/256, type=time_char, access=read/write>
In [4]: pv.get()                                                                                                                                         
Out[4]: 
array([ 82,  95,  48,  48,  57,  95,  99,  97, 109, 101, 114,  97,   0],
      dtype=uint8)
In [5]: pv.get(as_string=True)                                                                                                                           
Out[5]: 'R_009_camera'
prjemian commented 5 years ago

@newville : Same PyEpics version here: 3.3.1. You've used a different PV than my example. I confirm the PV you have used returns text for me, but the HDF1:XMLFileName PV is the problem here:

(base) user2bmb@lyra ~ $ ipython
Python 3.6.5 |Anaconda, Inc.| (default, Apr 29 2018, 16:14:56)
Type 'copyright', 'credits' or 'license' for more information
IPython 6.4.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import epics

In [2]: epics.__version__
Out[2]: '3.3.1'

In [3]: pv = epics.PV("2bmbPG3:TIFF1:FileName")

In [4]: pv
Out[4]: <PV '2bmbPG3:TIFF1:FileName', count=5/256, type=time_char, access=read/write>

In [5]: pv.get(as_string=True)
Out[5]: 'test'

In [6]: pv = epics.PV("2bmbPG3:HDF1:XMLFileName")

In [7]: pv
Out[7]: <PV '2bmbPG3:HDF1:XMLFileName', count=1048576/1048576, type=time_char, access=read/write>

In [8]: pv.get(as_string=True)
Out[8]: '<array size=15, type=time_char>'

The count attribute is a puzzle. Since you have identified a (seemingly) similar PV that behaves differently, it's time to compare the two EPICS databases where they are defined.

prjemian commented 5 years ago

from ADCore/ADApp/Db/NDFileHDF5.template (the count attribute is as-defined)

record(waveform, "$(P)$(R)XMLFileName")
{
    field(PINI, "YES")
    field(DTYP, "asynOctetWrite")
    field(INP,  "@asyn($(PORT),0)HDF5_layoutFilename")
    field(FTVL, "CHAR")
    field(NELM, "1048576")
    info(autosaveFields, "VAL")
}

record(waveform, "$(P)$(R)XMLFileName_RBV")
{
    field(DTYP, "asynOctetRead")
    field(INP,  "@asyn($(PORT),0)HDF5_layoutFilename")
    field(FTVL, "CHAR")
    field(NELM, "1048576")
    field(SCAN, "I/O Intr")
}

from ADCore/ADApp/Db/NDFile.template

record(waveform, "$(P)$(R)FileName")
{
    field(PINI, "YES")
    field(DTYP, "asynOctetWrite")
    field(INP,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))FILE_NAME")
    field(FTVL, "CHAR")
    field(NELM, "256")
    info(autosaveFields, "VAL")
}

record(waveform, "$(P)$(R)FileName_RBV")
{
    field(DTYP, "asynOctetRead")
    field(INP,  "@asyn($(PORT),$(ADDR=0),$(TIMEOUT=1))FILE_NAME")
    field(FTVL, "CHAR")
    field(NELM, "256")
    field(SCAN, "I/O Intr")
}
prjemian commented 5 years ago

@MarkRivers - in NDFileHDF5.template : XMLFileName, does the NELM field need to be so large?

newville commented 5 years ago

@prjemian Ah, sorry for not recognizing the important detail that 'XMLFileName' is > 16384. I can confirm the behavior you see in 3.3.1. This is fixed in git master and will be in 3.3.2.

prjemian commented 5 years ago

Problem is limited to this PV from area detector.

The PV was set to allow 1 MB text content so that the entire XML layout file could be communicated in this PV. The default size has been reduced in the AD source now.

The problem was that PyEpics 3.3.1 had a limit of 16 kB for waveform string content. That will be changed in the next version (3.3.2) to allow waveform string content larger than 16kB.

Once the PyEpics version is tested to resolve this, we can close this issue. Thanks @tacaswell , @newville , @MarkRivers!

prjemian commented 5 years ago

Note: the newer PyEpics code increases the length of arrays converted to string to 64 kB

newville commented 5 years ago

@prjemian The intention with https://github.com/pyepics/pyepics/pull/128 was that PV.get(as_string=True) would always convert a byte array to a string. But, it also appears that we do not have tests set up for converting very long CHAR waveforms. We'll add those.