BCDA-APS / use_bluesky

Tools to help APS use the Bluesky Framework (https://blueskyproject.io/)
8 stars 2 forks source link

how to connect HDF5 file plugin with instance of ADSimDetector #5

Closed prjemian closed 7 years ago

prjemian commented 7 years ago

Using the areaDetector (2.6) ADSimDetector with our planned tomo code. Don't know how to connect up the HDF5 file plugin with my instance of the SimDetector. The code below punts to PyEpics. There MUST be a better way. In ophyd.

from: https://github.com/BCDA-APS/use_bluesky/blob/master/plans/standlone.py#L100

from ophyd import SingleTrigger, AreaDetector, SimDetector

class MyDetector(SingleTrigger, SimDetector): pass

ad_prefix = '13SIM1:'
simdet = MyDetector(ad_prefix)
# assume sim detector is unconfigured, apply all config here
simdet.cam.acquire_time.put(0.5)    # seconds
simdet.cam.array_callbacks.put("Enable")
simdet.cam.data_type.put("UInt8")
simdet.cam.image_mode.put("Single")
simdet.cam.num_exposures.put(1)
simdet.cam.num_images.put(1)
simdet.cam.shutter_mode.put("None")
simdet.cam.trigger_mode.put("Internal")
# specific to sim detector
simdet.cam.sim_mode.put("LinearRamp")

# how to configure these plugins using ophyd?  use PyEpics for now
hdf5_prefix = simdet.name + 'HDF1:'
epics.caput(hdf5_prefix + 'EnableCallbacks', 'Enable')
epics.caput(hdf5_prefix + 'ArrayCallbacks', 'Enable')
epics.caput(hdf5_prefix + 'FilePath', os.getcwd())  # TODO: get from EPICS PV
epics.caput(hdf5_prefix + 'FileName', 'tomoscan')   # TODO: get from EPICS PV
epics.caput(hdf5_prefix + 'AutoIncrement', 'Yes')
epics.caput(hdf5_prefix + 'FileTemplate', '%s%s_%5.5d.h5')
epics.caput(hdf5_prefix + 'AutoSave', 'Yes')
epics.caput(hdf5_prefix + 'FileWriteMode', 'Single')
hdf5_frame_file = EpicsSignalRO(hdf5_prefix + 'FullFileName_RBV', name='hdf5_frame_file') 
tacaswell commented 7 years ago

Something like:

from ophyd import (EpicsSignalRO, EpicsSignal, Component as Cpt,
               DynamicDeviceComponent as DDCpt, Signal)
from ophyd import AreaDetector, SingleTrigger, HDF5Plugin, TIFFPlugin
from ophyd.areadetector.filestore_mixins import FileStoreHDF5IterativeWrite
from ophyd.areadetector.filestore_mixins import FileStoreTIFFIterativeWrite
from ophyd.areadetector import (ADComponent as ADCpt, EpicsSignalWithRBV,
                           ImagePlugin, StatsPlugin, DetectorBase,
SingleTrigger,ROIPlugin,ProcessPlugin,TransformPlugin)

class HDF5PluginWithFileStore(HDF5Plugin, FileStoreHDF5IterativeWrite):

    def get_frames_per_point(self):
        # return self.num_capture.get()
        return 1

class MyDetector(SingleTrigger, AreaDetector):

    image = Cpt(ImagePlugin, 'image1:')
    stats1 = Cpt(StatsPlugin, 'Stats1:')
    stats2 = Cpt(StatsPlugin, 'Stats2:')
    stats3 = Cpt(StatsPlugin, 'Stats3:')
    stats4 = Cpt(StatsPlugin, 'Stats4:')
    stats5 = Cpt(StatsPlugin, 'Stats5:')
    trans1 = Cpt(TransformPlugin, 'Trans1:')
    roi1 = Cpt(ROIPlugin, 'ROI1:')
    roi2 = Cpt(ROIPlugin, 'ROI2:')
    roi3 = Cpt(ROIPlugin, 'ROI3:')
    roi4 = Cpt(ROIPlugin, 'ROI4:')
    proc1 = Cpt(ProcessPlugin, 'Proc1:')

    hdf5 = Cpt(HDF5PluginWithFileStore,
               suffix='HDF1:',
               write_path_template='/direct/XF21ID1/image_files/',  # trailing slash!
               read_path_template='/direct/XF21ID1/image_files/')

Was a bit over-agressive about copying imports, should work though.

If you are using master, you can pass fs=file_store_instance in as a kwarg to the hdf5 plugin.

On stage this will generate a unique filename and inserts the correct values into filestore. The 'iterative' part of the class name is because there is a version that writes to filestore on every trigger and a version that does a bulk insert of all of them at the end.

prjemian commented 7 years ago

Skipping the FileStore part first, just enhancing the MyDetector class and stumped early. @tacaswell @danielballan, Can either of you assist? Look in the "add-HDF5=handling" branch.

With this definition, the scan plan progresses to completion.

class MyDetector(SimDetector, SingleTrigger):

    #image1 = Cpt(ImagePlugin, 'image1:')
    #hdf1 = Cpt(myHDF5Plugin, 'HDF1:')
    no_op = None

Defining the image1 attribute, the scan plan fails:

class MyDetector(SimDetector, SingleTrigger):

    image1 = Cpt(ImagePlugin, 'image1:')
    #hdf1 = Cpt(myHDF5Plugin, 'HDF1:')
    no_op = None

Here's the console log with exception trace:

Loading metadata history from /home/prjemian/.config/bluesky/bluesky_history.db
Transient Scan ID: 140
Persistent Unique Scan ID: '3bf89a1c-78e4-45a1-b36e-3658f5058961'
/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/epics/ca.py:1248: UserWarning: ca.get('13SIM1:image1:ArrayData') timed out after 1.00 seconds.
  warnings.warn(msg % (name(chid), timeout))
/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/epics/ca.py:1248: UserWarning: ca.get('13SIM1:image1:ArrayData') timed out after 8.08 seconds.
  warnings.warn(msg % (name(chid), timeout))
generator interlace_tomo_scan ['3bf89a'] (scan num: 140)
Traceback (most recent call last):
  File "/home/prjemian/Documents/eclipse/use_bluesky/plans/standlone.py", line 311, in <module>
    RE(tomo_plan, tomo_callbacks, md=dict(developer=True))
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 599, in __call__
    raise exc
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 1007, in _run
    raise err
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 905, in _run
    msg = self._plan_stack[-1].send(resp)
  File "/home/prjemian/Documents/eclipse/use_bluesky/plans/interlace_tomo.py", line 179, in interlace_tomo_scan
    return (yield from inner_scan())
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 47, in dec_inner
    return (yield from plan)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1510, in stage_wrapper
    return (yield from finalize_wrapper(inner(), unstage_devices()))
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1025, in finalize_wrapper
    ret = yield from plan
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1508, in inner
    return (yield from plan)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 47, in dec_inner
    return (yield from plan)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 873, in run_wrapper
    yield from plan
  File "/home/prjemian/Documents/eclipse/use_bluesky/plans/interlace_tomo.py", line 177, in inner_scan
    yield from per_step(detectors, motor, projection)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 2010, in one_1d_step
    return (yield from trigger_and_read(list(detectors) + [motor]))
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1819, in trigger_and_read
    return (yield from rewindable_wrapper(pchain(*plan_stack), rewindable))
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1222, in rewindable_wrapper
    return (yield from plan)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 288, in pchain
    rets.append((yield from p))
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/utils.py", line 122, in single_gen
    return (yield msg)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 956, in _run
    response = yield from coro(msg)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/asyncio/coroutines.py", line 206, in coro
    res = func(*args, **kw)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 1269, in _read
    data_keys = obj.describe()
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/device.py", line 728, in describe
    res.update(self._describe_attr_list(self.read_attrs))
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/device.py", line 721, in _describe_attr_list
    desc.update(obj.describe())
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/device.py", line 728, in describe
    res.update(self._describe_attr_list(self.read_attrs))
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/device.py", line 721, in _describe_attr_list
    desc.update(obj.describe())
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/signal.py", line 570, in describe
    desc['dtype'] = data_type(val)
  File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/utils/epics_pvs.py", line 464, in data_type
    raise ValueError('{} not a valid type (int, float, ndarray, str)'.format(val))
ValueError: None not a valid type (int, float, ndarray, str)
danielballan commented 7 years ago

I don't remember seeing that particular ValueError before. It seems like some PVs may not be correct and that that is causing secondary problems further down. I notice in particular the line:

/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/epics/ca.py:1248: UserWarning: ca.get('13SIM1:image1:ArrayData') timed out after 8.08 seconds.

Before attempting to scan, let's make sure the device is fully connected by accessing detector_instance.connected. (From your traceback I can't tell what you called detector_instance, the instance of MyDetector.) If that's False, then test each component in turn, such as detector_instance.image1.connected.

tacaswell commented 7 years ago

What simioc are you running against? Would you happen to have a docker image with it in it?

prjemian commented 7 years ago

Trying to work from the latest release of AD, it has the best version of the HDF file plugin. AD 2.5 introduced the major changes in the HDF5 plugin that were not working correctly in AD 2.4. 2.6 handled some new aspects, as well as provided code to handle the HDF5 SWMR feature (that I am not using). The earliest SWMR release had some problems which have been resolved recently and will be interfaced in the next AD release.

prjemian@ookhd .../iocBoot/iocSimDetector $ pwd
/opt/epics/synApps/support/areaDetector-R2-6/ADSimDetector-R2-4/iocs/simDetectorIOC/iocBoot/iocSimDetector
prjemian@ookhd .../iocBoot/iocSimDetector $ uname -a
Linux ookhd 4.4.0-72-generic #93-Ubuntu SMP Fri Mar 31 14:07:41 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

I don't have a Docker image.

prjemian commented 7 years ago

@danielballan : the instance is called simdet at the global level and is hosted on the same linux computer as the BlueSky session. I can watch the areaDetector GUI controls and observe what is happening so I know the AD IOC is connected and functioning.

prjemian commented 7 years ago

Digging somewhat deeper this morning, I find this in ophyd/areadetector/plugins.py: class FilePlugin

file_number_sync = C(EpicsSignal, 'FileNumber_Sync')

In the IOC's shell, I make a file listing all the known PVs:

dbl > dbl-all.txt

then switch to a linux console and look for all PVs that might match:

prjemian@ookhd .../iocBoot/iocSimDetector $ grep FileNumber dbl-all.txt 
13SIM1:netCDF1:FileNumber
13SIM1:TIFF1:FileNumber
13SIM1:JPEG1:FileNumber
13SIM1:Nexus1:FileNumber
13SIM1:HDF1:FileNumber
13SIM1:Magick1:FileNumber
13SIM1:netCDF1:FileNumber_RBV
13SIM1:TIFF1:FileNumber_RBV
13SIM1:JPEG1:FileNumber_RBV
13SIM1:Nexus1:FileNumber_RBV
13SIM1:HDF1:FileNumber_RBV
13SIM1:Magick1:FileNumber_RBV

(same directory as given above, the IOC's boot directory)

There is no such FileNumber_Sync PV. That's one problem. Not sure if it is the one that caused the ValueError. I suspect that is a different problem.

prjemian commented 7 years ago

The problem with FileNumber_Sync was found while subclassing HDF5Plugin.

class myHDF5Plugin(Device):
    # can't sublclass HDF5Plugin, that expects a FileNumber_Sync PV that does not exist
    # see: ophyd/areadetector/plugins.py: class FilePlugin

    array_callbacks = Cpt(EpicsSignalWithRBV, 'ArrayCallbacks')
    enable_callbacks = Cpt(EpicsSignalWithRBV, 'EnableCallbacks')
    auto_increment = Cpt(EpicsSignalWithRBV, 'AutoIncrement')
    auto_save = Cpt(EpicsSignalWithRBV, 'AutoSave')
    file_path = Cpt(EpicsSignalWithRBV, 'FilePath', string=True)
    file_name = Cpt(EpicsSignalWithRBV, 'FileName', string=True)
    file_number = Cpt(EpicsSignalWithRBV, 'FileNumber')
    file_template = Cpt(EpicsSignalWithRBV, 'FileTemplate', string=True)
    file_write_mode = Cpt(EpicsSignalWithRBV, 'FileWriteMode')
    full_file_name = Cpt(EpicsSignalRO, 'FullFileName_RBV', string=True)
    xml_layout_file = Cpt(EpicsSignalWithRBV, 'XMLFileName', string=True)

class MyDetector(SimDetector, SingleTrigger):

    #image1 = Cpt(ImagePlugin, 'image1:')
    hdf1 = Cpt(myHDF5Plugin, 'HDF1:')
    no_op = None

As the comment shows, the HDF5Plugin expects a FileNumber_Sync PV that does not exist

MarkRivers commented 7 years ago

FileNumber_Sync was removed in this commit to NDFile.template

commit 6fb231f5269adcbea01a93247a45255057577c07
Author: Mark Rivers <rivers@cars.uchicago.edu>
Date:   Thu Jan 8 10:35:29 2015 -0600

    Changed FileNumber to use asyn:READBACK info tag for updating longout record via callbacks, rather than ugly database logic

Note that beginning with R2-6 all plugins and drivers now have the following records:

###################################################################
#  This record contains the version of ADCore                     #
###################################################################

record(stringin, "$(P)$(R)ADCoreVersion_RBV")
{
   field(DTYP, "asynOctetRead")
   field(INP,  "@asyn($(PORT),$(ADDR),$(TIMEOUT))ADCORE_VERSION")
   field(VAL,  "Unknown")
   field(SCAN, "I/O Intr")
}

###################################################################
#  This record contains the version of the driver or plugin       #
###################################################################

record(stringin, "$(P)$(R)DriverVersion_RBV")
{
   field(DTYP, "asynOctetRead")
   field(INP,  "@asyn($(PORT),$(ADDR),$(TIMEOUT))DRIVER_VERSION")
   field(VAL,  "Unknown")
   field(SCAN, "I/O Intr")
}

These will allow clients like Python to determine which version of ADCore and of the driver/plugin is being used, and so can change their behavior accordingly. Unfortunately for releases prior to R2-6 this is not possible, and so it will require searching for a PV to see if it exists.

danielballan commented 7 years ago

Sorry, we have seen that problem on the floor here but failed to document it properly. We should probably package different versions of our AD ophyd devices for different versions AD.

I do think your ValueError problem is unrelated. Circling back to your last comment on that, I think it would still be useful to check simdet.connected (even if the GUI controls report that the IOC is fine) to verify that, for example, all the PVs are spelled correctly.

tacaswell commented 7 years ago

Locally we use a mix of high 1.x and low 2.x AreaDetector installations and the classes are tuned to the PVs that those provide.

def recur_pv(obj):
    try:
        print(obj.name, obj.pvname)
    except AttributeError:
        if not hasattr(obj, 'signal_names'):
            return
        for c in obj.signal_names:
            recur_pv(getattr(obj, c))

might be a useful bit of code to get all of the (read) PVs that the object thinks it knows about.

which gives something like ``` In [14]: det = MyDetector(prefix, name='det') In [15]: recur_pv(det) det_cam_array_counter XF:31IDA-BI{Cam:Tbl}cam1:ArrayCounter_RBV det_cam_array_rate XF:31IDA-BI{Cam:Tbl}cam1:ArrayRate_RBV det_cam_asyn_io XF:31IDA-BI{Cam:Tbl}cam1:AsynIO det_cam_nd_attributes_file XF:31IDA-BI{Cam:Tbl}cam1:NDAttributesFile det_cam_pool_alloc_buffers XF:31IDA-BI{Cam:Tbl}cam1:PoolAllocBuffers det_cam_pool_free_buffers XF:31IDA-BI{Cam:Tbl}cam1:PoolFreeBuffers det_cam_pool_max_buffers XF:31IDA-BI{Cam:Tbl}cam1:PoolMaxBuffers det_cam_pool_max_mem XF:31IDA-BI{Cam:Tbl}cam1:PoolMaxMem det_cam_pool_used_buffers XF:31IDA-BI{Cam:Tbl}cam1:PoolUsedBuffers det_cam_pool_used_mem XF:31IDA-BI{Cam:Tbl}cam1:PoolUsedMem det_cam_port_name XF:31IDA-BI{Cam:Tbl}cam1:PortName_RBV det_cam_acquire XF:31IDA-BI{Cam:Tbl}cam1:Acquire_RBV det_cam_acquire_period XF:31IDA-BI{Cam:Tbl}cam1:AcquirePeriod_RBV det_cam_acquire_time XF:31IDA-BI{Cam:Tbl}cam1:AcquireTime_RBV det_cam_array_callbacks XF:31IDA-BI{Cam:Tbl}cam1:ArrayCallbacks_RBV det_cam_array_size_array_size_x XF:31IDA-BI{Cam:Tbl}cam1:ArraySizeX_RBV det_cam_array_size_array_size_y XF:31IDA-BI{Cam:Tbl}cam1:ArraySizeY_RBV det_cam_array_size_array_size_z XF:31IDA-BI{Cam:Tbl}cam1:ArraySizeZ_RBV det_cam_array_size_bytes XF:31IDA-BI{Cam:Tbl}cam1:ArraySize_RBV det_cam_bin_x XF:31IDA-BI{Cam:Tbl}cam1:BinX_RBV det_cam_bin_y XF:31IDA-BI{Cam:Tbl}cam1:BinY_RBV det_cam_color_mode XF:31IDA-BI{Cam:Tbl}cam1:ColorMode_RBV det_cam_data_type XF:31IDA-BI{Cam:Tbl}cam1:DataType_RBV det_cam_detector_state XF:31IDA-BI{Cam:Tbl}cam1:DetectorState_RBV det_cam_frame_type XF:31IDA-BI{Cam:Tbl}cam1:FrameType_RBV det_cam_gain XF:31IDA-BI{Cam:Tbl}cam1:Gain_RBV det_cam_image_mode XF:31IDA-BI{Cam:Tbl}cam1:ImageMode_RBV det_cam_manufacturer XF:31IDA-BI{Cam:Tbl}cam1:Manufacturer_RBV det_cam_max_size_max_size_x XF:31IDA-BI{Cam:Tbl}cam1:MaxSizeX_RBV det_cam_max_size_max_size_y XF:31IDA-BI{Cam:Tbl}cam1:MaxSizeY_RBV det_cam_min_x XF:31IDA-BI{Cam:Tbl}cam1:MinX_RBV det_cam_min_y XF:31IDA-BI{Cam:Tbl}cam1:MinY_RBV det_cam_model XF:31IDA-BI{Cam:Tbl}cam1:Model_RBV det_cam_num_exposures XF:31IDA-BI{Cam:Tbl}cam1:NumExposures_RBV det_cam_num_exposures_counter XF:31IDA-BI{Cam:Tbl}cam1:NumExposuresCounter_RBV det_cam_num_images XF:31IDA-BI{Cam:Tbl}cam1:NumImages_RBV det_cam_num_images_counter XF:31IDA-BI{Cam:Tbl}cam1:NumImagesCounter_RBV det_cam_read_status XF:31IDA-BI{Cam:Tbl}cam1:ReadStatus det_cam_reverse_reverse_x XF:31IDA-BI{Cam:Tbl}cam1:ReverseX_RBV det_cam_reverse_reverse_y XF:31IDA-BI{Cam:Tbl}cam1:ReverseY_RBV det_cam_shutter_close_delay XF:31IDA-BI{Cam:Tbl}cam1:ShutterCloseDelay_RBV det_cam_shutter_close_epics XF:31IDA-BI{Cam:Tbl}cam1:ShutterCloseEPICS det_cam_shutter_control XF:31IDA-BI{Cam:Tbl}cam1:ShutterControl_RBV det_cam_shutter_control_epics XF:31IDA-BI{Cam:Tbl}cam1:ShutterControlEPICS det_cam_shutter_fanout XF:31IDA-BI{Cam:Tbl}cam1:ShutterFanout det_cam_shutter_mode XF:31IDA-BI{Cam:Tbl}cam1:ShutterMode_RBV det_cam_shutter_open_delay XF:31IDA-BI{Cam:Tbl}cam1:ShutterOpenDelay_RBV det_cam_shutter_open_epics XF:31IDA-BI{Cam:Tbl}cam1:ShutterOpenEPICS det_cam_shutter_status_epics XF:31IDA-BI{Cam:Tbl}cam1:ShutterStatusEPICS_RBV det_cam_shutter_status XF:31IDA-BI{Cam:Tbl}cam1:ShutterStatus_RBV det_cam_size_size_x XF:31IDA-BI{Cam:Tbl}cam1:SizeX_RBV det_cam_size_size_y XF:31IDA-BI{Cam:Tbl}cam1:SizeY_RBV det_cam_status_message XF:31IDA-BI{Cam:Tbl}cam1:StatusMessage_RBV det_cam_string_from_server XF:31IDA-BI{Cam:Tbl}cam1:StringFromServer_RBV det_cam_string_to_server XF:31IDA-BI{Cam:Tbl}cam1:StringToServer_RBV det_cam_temperature XF:31IDA-BI{Cam:Tbl}cam1:Temperature_RBV det_cam_temperature_actual XF:31IDA-BI{Cam:Tbl}cam1:TemperatureActual det_cam_time_remaining XF:31IDA-BI{Cam:Tbl}cam1:TimeRemaining_RBV det_cam_trigger_mode XF:31IDA-BI{Cam:Tbl}cam1:TriggerMode_RBV det_cam_gain_rgb_gain_red XF:31IDA-BI{Cam:Tbl}cam1:GainRed_RBV det_cam_gain_rgb_gain_green XF:31IDA-BI{Cam:Tbl}cam1:GainGreen_RBV det_cam_gain_rgb_gain_blue XF:31IDA-BI{Cam:Tbl}cam1:GainBlue_RBV det_cam_gain_xy_gain_x XF:31IDA-BI{Cam:Tbl}cam1:GainX_RBV det_cam_gain_xy_gain_y XF:31IDA-BI{Cam:Tbl}cam1:GainY_RBV det_cam_noise XF:31IDA-BI{Cam:Tbl}cam1:Noise_RBV det_cam_peak_num_peak_num_x XF:31IDA-BI{Cam:Tbl}cam1:PeakNumX_RBV det_cam_peak_num_peak_num_y XF:31IDA-BI{Cam:Tbl}cam1:PeakNumY_RBV det_cam_peak_start_peak_start_x XF:31IDA-BI{Cam:Tbl}cam1:PeakStartX_RBV det_cam_peak_start_peak_start_y XF:31IDA-BI{Cam:Tbl}cam1:PeakStartY_RBV det_cam_peak_step_peak_step_x XF:31IDA-BI{Cam:Tbl}cam1:PeakStepX_RBV det_cam_peak_step_peak_step_y XF:31IDA-BI{Cam:Tbl}cam1:PeakStepY_RBV det_cam_peak_variation XF:31IDA-BI{Cam:Tbl}cam1:PeakVariation_RBV det_cam_peak_width_peak_width_x XF:31IDA-BI{Cam:Tbl}cam1:PeakWidthX_RBV det_cam_peak_width_peak_width_y XF:31IDA-BI{Cam:Tbl}cam1:PeakWidthY_RBV det_cam_reset XF:31IDA-BI{Cam:Tbl}cam1:Reset_RBV det_cam_sim_mode XF:31IDA-BI{Cam:Tbl}cam1:SimMode_RBV det_tiff1_array_counter XF:31IDA-BI{Cam:Tbl}TIFF1:ArrayCounter_RBV det_tiff1_array_rate XF:31IDA-BI{Cam:Tbl}TIFF1:ArrayRate_RBV det_tiff1_asyn_io XF:31IDA-BI{Cam:Tbl}TIFF1:AsynIO det_tiff1_nd_attributes_file XF:31IDA-BI{Cam:Tbl}TIFF1:NDAttributesFile det_tiff1_pool_alloc_buffers XF:31IDA-BI{Cam:Tbl}TIFF1:PoolAllocBuffers det_tiff1_pool_free_buffers XF:31IDA-BI{Cam:Tbl}TIFF1:PoolFreeBuffers det_tiff1_pool_max_buffers XF:31IDA-BI{Cam:Tbl}TIFF1:PoolMaxBuffers det_tiff1_pool_max_mem XF:31IDA-BI{Cam:Tbl}TIFF1:PoolMaxMem det_tiff1_pool_used_buffers XF:31IDA-BI{Cam:Tbl}TIFF1:PoolUsedBuffers det_tiff1_pool_used_mem XF:31IDA-BI{Cam:Tbl}TIFF1:PoolUsedMem det_tiff1_port_name XF:31IDA-BI{Cam:Tbl}TIFF1:PortName_RBV det_tiff1_width XF:31IDA-BI{Cam:Tbl}TIFF1:ArraySize0_RBV det_tiff1_height XF:31IDA-BI{Cam:Tbl}TIFF1:ArraySize1_RBV det_tiff1_depth XF:31IDA-BI{Cam:Tbl}TIFF1:ArraySize2_RBV det_tiff1_array_size_height XF:31IDA-BI{Cam:Tbl}TIFF1:ArraySize1_RBV det_tiff1_array_size_width XF:31IDA-BI{Cam:Tbl}TIFF1:ArraySize0_RBV det_tiff1_array_size_depth XF:31IDA-BI{Cam:Tbl}TIFF1:ArraySize2_RBV det_tiff1_bayer_pattern XF:31IDA-BI{Cam:Tbl}TIFF1:BayerPattern_RBV det_tiff1_blocking_callbacks XF:31IDA-BI{Cam:Tbl}TIFF1:BlockingCallbacks_RBV det_tiff1_color_mode XF:31IDA-BI{Cam:Tbl}TIFF1:ColorMode_RBV det_tiff1_data_type XF:31IDA-BI{Cam:Tbl}TIFF1:DataType_RBV det_tiff1_dim0_sa XF:31IDA-BI{Cam:Tbl}TIFF1:Dim0SA det_tiff1_dim1_sa XF:31IDA-BI{Cam:Tbl}TIFF1:Dim1SA det_tiff1_dim2_sa XF:31IDA-BI{Cam:Tbl}TIFF1:Dim2SA det_tiff1_dim_sa_dim0 XF:31IDA-BI{Cam:Tbl}TIFF1:Dim0SA det_tiff1_dim_sa_dim1 XF:31IDA-BI{Cam:Tbl}TIFF1:Dim1SA det_tiff1_dim_sa_dim2 XF:31IDA-BI{Cam:Tbl}TIFF1:Dim2SA det_tiff1_dimensions XF:31IDA-BI{Cam:Tbl}TIFF1:Dimensions_RBV det_tiff1_dropped_arrays XF:31IDA-BI{Cam:Tbl}TIFF1:DroppedArrays_RBV det_tiff1_enable XF:31IDA-BI{Cam:Tbl}TIFF1:EnableCallbacks_RBV det_tiff1_min_callback_time XF:31IDA-BI{Cam:Tbl}TIFF1:MinCallbackTime_RBV det_tiff1_nd_array_address XF:31IDA-BI{Cam:Tbl}TIFF1:NDArrayAddress_RBV det_tiff1_nd_array_port XF:31IDA-BI{Cam:Tbl}TIFF1:NDArrayPort_RBV det_tiff1_ndimensions XF:31IDA-BI{Cam:Tbl}TIFF1:NDimensions_RBV det_tiff1_plugin_type XF:31IDA-BI{Cam:Tbl}TIFF1:PluginType_RBV det_tiff1_queue_free XF:31IDA-BI{Cam:Tbl}TIFF1:QueueFree det_tiff1_queue_free_low XF:31IDA-BI{Cam:Tbl}TIFF1:QueueFreeLow det_tiff1_queue_size XF:31IDA-BI{Cam:Tbl}TIFF1:QueueSize det_tiff1_queue_use XF:31IDA-BI{Cam:Tbl}TIFF1:QueueUse det_tiff1_queue_use_high XF:31IDA-BI{Cam:Tbl}TIFF1:QueueUseHIGH det_tiff1_queue_use_hihi XF:31IDA-BI{Cam:Tbl}TIFF1:QueueUseHIHI det_tiff1_time_stamp XF:31IDA-BI{Cam:Tbl}TIFF1:TimeStamp_RBV det_tiff1_unique_id XF:31IDA-BI{Cam:Tbl}TIFF1:UniqueId_RBV det_tiff1_auto_increment XF:31IDA-BI{Cam:Tbl}TIFF1:AutoIncrement_RBV det_tiff1_auto_save XF:31IDA-BI{Cam:Tbl}TIFF1:AutoSave_RBV det_tiff1_capture XF:31IDA-BI{Cam:Tbl}TIFF1:Capture_RBV det_tiff1_delete_driver_file XF:31IDA-BI{Cam:Tbl}TIFF1:DeleteDriverFile_RBV det_tiff1_file_format XF:31IDA-BI{Cam:Tbl}TIFF1:FileFormat_RBV det_tiff1_file_name XF:31IDA-BI{Cam:Tbl}TIFF1:FileName_RBV det_tiff1_file_number XF:31IDA-BI{Cam:Tbl}TIFF1:FileNumber_RBV det_tiff1_file_number_sync XF:31IDA-BI{Cam:Tbl}TIFF1:FileNumber_Sync det_tiff1_file_number_write XF:31IDA-BI{Cam:Tbl}TIFF1:FileNumber_write det_tiff1_file_path XF:31IDA-BI{Cam:Tbl}TIFF1:FilePath_RBV det_tiff1_file_path_exists XF:31IDA-BI{Cam:Tbl}TIFF1:FilePathExists_RBV det_tiff1_file_template XF:31IDA-BI{Cam:Tbl}TIFF1:FileTemplate_RBV det_tiff1_file_write_mode XF:31IDA-BI{Cam:Tbl}TIFF1:FileWriteMode_RBV det_tiff1_full_file_name XF:31IDA-BI{Cam:Tbl}TIFF1:FullFileName_RBV det_tiff1_num_capture XF:31IDA-BI{Cam:Tbl}TIFF1:NumCapture_RBV det_tiff1_num_captured XF:31IDA-BI{Cam:Tbl}TIFF1:NumCaptured_RBV det_tiff1_read_file XF:31IDA-BI{Cam:Tbl}TIFF1:ReadFile_RBV det_tiff1_write_file XF:31IDA-BI{Cam:Tbl}TIFF1:WriteFile_RBV det_tiff1_write_message XF:31IDA-BI{Cam:Tbl}TIFF1:WriteMessage det_tiff1_write_status XF:31IDA-BI{Cam:Tbl}TIFF1:WriteStatus ``` as output
MarkRivers commented 7 years ago

Is Python actually using FileNumber_Sync?

That PV was never intended to be part of the public API for areaDetector. Note that it was never documented in areaDetectorDoc.html, and was never exposed in any of the OPI files (medm, css, etc.). It was only intended as a database hack to update FileNumber whenever a callback updated FileNumber_RBV.

prjemian commented 7 years ago

added

_det = simdet.connected

debugger shows

_det  bool: True
prjemian commented 7 years ago

That is True for both variations:

class MyDetector(SimDetector, SingleTrigger):

    image1 = Cpt(ImagePlugin, 'image1:')
    no_op = None

and

class MyDetector(SimDetector, SingleTrigger):

    # image1 = Cpt(ImagePlugin, 'image1:')
    no_op = None
prjemian commented 7 years ago

Looking at the ValueError trace above, how can I find what has triggered the exception? At some point (after the start document), it looks as if there is {} in the json string that is reported.

prjemian commented 7 years ago

Applying the recur_pv() function to me session:

simdet = MyDetector(AD_PREFIX)
setupSimDetector(simdet)
print('area detector connected:', simdet.connected)
recur_pv(simdet)

Here is the session output:

pydev debugger: starting (pid: 5948)
Loading metadata history from /home/prjemian/.config/bluesky/bluesky_history.db
area detector connected: True
13SIM1:_cam_array_counter 13SIM1:cam1:ArrayCounter_RBV
13SIM1:_cam_array_rate 13SIM1:cam1:ArrayRate_RBV
13SIM1:_cam_asyn_io 13SIM1:cam1:AsynIO
13SIM1:_cam_nd_attributes_file 13SIM1:cam1:NDAttributesFile
13SIM1:_cam_pool_alloc_buffers 13SIM1:cam1:PoolAllocBuffers
13SIM1:_cam_pool_free_buffers 13SIM1:cam1:PoolFreeBuffers
13SIM1:_cam_pool_max_buffers 13SIM1:cam1:PoolMaxBuffers
13SIM1:_cam_pool_max_mem 13SIM1:cam1:PoolMaxMem
13SIM1:_cam_pool_used_buffers 13SIM1:cam1:PoolUsedBuffers
13SIM1:_cam_pool_used_mem 13SIM1:cam1:PoolUsedMem
13SIM1:_cam_port_name 13SIM1:cam1:PortName_RBV
13SIM1:_cam_acquire 13SIM1:cam1:Acquire_RBV
13SIM1:_cam_acquire_period 13SIM1:cam1:AcquirePeriod_RBV
13SIM1:_cam_acquire_time 13SIM1:cam1:AcquireTime_RBV
13SIM1:_cam_array_callbacks 13SIM1:cam1:ArrayCallbacks_RBV
13SIM1:_cam_array_size_array_size_x 13SIM1:cam1:ArraySizeX_RBV
13SIM1:_cam_array_size_array_size_y 13SIM1:cam1:ArraySizeY_RBV
13SIM1:_cam_array_size_array_size_z 13SIM1:cam1:ArraySizeZ_RBV
13SIM1:_cam_array_size_bytes 13SIM1:cam1:ArraySize_RBV
13SIM1:_cam_bin_x 13SIM1:cam1:BinX_RBV
13SIM1:_cam_bin_y 13SIM1:cam1:BinY_RBV
13SIM1:_cam_color_mode 13SIM1:cam1:ColorMode_RBV
13SIM1:_cam_data_type 13SIM1:cam1:DataType_RBV
13SIM1:_cam_detector_state 13SIM1:cam1:DetectorState_RBV
13SIM1:_cam_frame_type 13SIM1:cam1:FrameType_RBV
13SIM1:_cam_gain 13SIM1:cam1:Gain_RBV
13SIM1:_cam_image_mode 13SIM1:cam1:ImageMode_RBV
13SIM1:_cam_manufacturer 13SIM1:cam1:Manufacturer_RBV
13SIM1:_cam_max_size_max_size_x 13SIM1:cam1:MaxSizeX_RBV
13SIM1:_cam_max_size_max_size_y 13SIM1:cam1:MaxSizeY_RBV
13SIM1:_cam_min_x 13SIM1:cam1:MinX_RBV
13SIM1:_cam_min_y 13SIM1:cam1:MinY_RBV
13SIM1:_cam_model 13SIM1:cam1:Model_RBV
13SIM1:_cam_num_exposures 13SIM1:cam1:NumExposures_RBV
13SIM1:_cam_num_exposures_counter 13SIM1:cam1:NumExposuresCounter_RBV
13SIM1:_cam_num_images 13SIM1:cam1:NumImages_RBV
13SIM1:_cam_num_images_counter 13SIM1:cam1:NumImagesCounter_RBV
13SIM1:_cam_read_status 13SIM1:cam1:ReadStatus
13SIM1:_cam_reverse_reverse_x 13SIM1:cam1:ReverseX_RBV
13SIM1:_cam_reverse_reverse_y 13SIM1:cam1:ReverseY_RBV
13SIM1:_cam_shutter_close_delay 13SIM1:cam1:ShutterCloseDelay_RBV
13SIM1:_cam_shutter_close_epics 13SIM1:cam1:ShutterCloseEPICS
13SIM1:_cam_shutter_control 13SIM1:cam1:ShutterControl_RBV
13SIM1:_cam_shutter_control_epics 13SIM1:cam1:ShutterControlEPICS
13SIM1:_cam_shutter_fanout 13SIM1:cam1:ShutterFanout
13SIM1:_cam_shutter_mode 13SIM1:cam1:ShutterMode_RBV
13SIM1:_cam_shutter_open_delay 13SIM1:cam1:ShutterOpenDelay_RBV
13SIM1:_cam_shutter_open_epics 13SIM1:cam1:ShutterOpenEPICS
13SIM1:_cam_shutter_status_epics 13SIM1:cam1:ShutterStatusEPICS_RBV
13SIM1:_cam_shutter_status 13SIM1:cam1:ShutterStatus_RBV
13SIM1:_cam_size_size_x 13SIM1:cam1:SizeX_RBV
13SIM1:_cam_size_size_y 13SIM1:cam1:SizeY_RBV
13SIM1:_cam_status_message 13SIM1:cam1:StatusMessage_RBV
13SIM1:_cam_string_from_server 13SIM1:cam1:StringFromServer_RBV
13SIM1:_cam_string_to_server 13SIM1:cam1:StringToServer_RBV
13SIM1:_cam_temperature 13SIM1:cam1:Temperature_RBV
13SIM1:_cam_temperature_actual 13SIM1:cam1:TemperatureActual
13SIM1:_cam_time_remaining 13SIM1:cam1:TimeRemaining_RBV
13SIM1:_cam_trigger_mode 13SIM1:cam1:TriggerMode_RBV
13SIM1:_cam_gain_rgb_gain_red 13SIM1:cam1:GainRed_RBV
13SIM1:_cam_gain_rgb_gain_green 13SIM1:cam1:GainGreen_RBV
13SIM1:_cam_gain_rgb_gain_blue 13SIM1:cam1:GainBlue_RBV
13SIM1:_cam_gain_xy_gain_x 13SIM1:cam1:GainX_RBV
13SIM1:_cam_gain_xy_gain_y 13SIM1:cam1:GainY_RBV
13SIM1:_cam_noise 13SIM1:cam1:Noise_RBV
13SIM1:_cam_peak_num_peak_num_x 13SIM1:cam1:PeakNumX_RBV
13SIM1:_cam_peak_num_peak_num_y 13SIM1:cam1:PeakNumY_RBV
13SIM1:_cam_peak_start_peak_start_x 13SIM1:cam1:PeakStartX_RBV
13SIM1:_cam_peak_start_peak_start_y 13SIM1:cam1:PeakStartY_RBV
13SIM1:_cam_peak_step_peak_step_x 13SIM1:cam1:PeakStepX_RBV
13SIM1:_cam_peak_step_peak_step_y 13SIM1:cam1:PeakStepY_RBV
13SIM1:_cam_peak_variation 13SIM1:cam1:PeakVariation_RBV
13SIM1:_cam_peak_width_peak_width_x 13SIM1:cam1:PeakWidthX_RBV
13SIM1:_cam_peak_width_peak_width_y 13SIM1:cam1:PeakWidthY_RBV
13SIM1:_cam_reset 13SIM1:cam1:Reset_RBV
13SIM1:_cam_sim_mode 13SIM1:cam1:SimMode_RBV
13SIM1:_image1_array_counter 13SIM1:image1:ArrayCounter_RBV
13SIM1:_image1_array_rate 13SIM1:image1:ArrayRate_RBV
13SIM1:_image1_asyn_io 13SIM1:image1:AsynIO
13SIM1:_image1_nd_attributes_file 13SIM1:image1:NDAttributesFile
13SIM1:_image1_pool_alloc_buffers 13SIM1:image1:PoolAllocBuffers
13SIM1:_image1_pool_free_buffers 13SIM1:image1:PoolFreeBuffers
13SIM1:_image1_pool_max_buffers 13SIM1:image1:PoolMaxBuffers
13SIM1:_image1_pool_max_mem 13SIM1:image1:PoolMaxMem
13SIM1:_image1_pool_used_buffers 13SIM1:image1:PoolUsedBuffers
13SIM1:_image1_pool_used_mem 13SIM1:image1:PoolUsedMem
13SIM1:_image1_port_name 13SIM1:image1:PortName_RBV
13SIM1:_image1_width 13SIM1:image1:ArraySize0_RBV
13SIM1:_image1_height 13SIM1:image1:ArraySize1_RBV
13SIM1:_image1_depth 13SIM1:image1:ArraySize2_RBV
13SIM1:_image1_array_size_height 13SIM1:image1:ArraySize1_RBV
13SIM1:_image1_array_size_width 13SIM1:image1:ArraySize0_RBV
13SIM1:_image1_array_size_depth 13SIM1:image1:ArraySize2_RBV
13SIM1:_image1_bayer_pattern 13SIM1:image1:BayerPattern_RBV
13SIM1:_image1_blocking_callbacks 13SIM1:image1:BlockingCallbacks_RBV
13SIM1:_image1_color_mode 13SIM1:image1:ColorMode_RBV
13SIM1:_image1_data_type 13SIM1:image1:DataType_RBV
13SIM1:_image1_dim0_sa 13SIM1:image1:Dim0SA
13SIM1:_image1_dim1_sa 13SIM1:image1:Dim1SA
13SIM1:_image1_dim2_sa 13SIM1:image1:Dim2SA
13SIM1:_image1_dim_sa_dim0 13SIM1:image1:Dim0SA
13SIM1:_image1_dim_sa_dim1 13SIM1:image1:Dim1SA
13SIM1:_image1_dim_sa_dim2 13SIM1:image1:Dim2SA
13SIM1:_image1_dimensions 13SIM1:image1:Dimensions_RBV
13SIM1:_image1_dropped_arrays 13SIM1:image1:DroppedArrays_RBV
13SIM1:_image1_enable 13SIM1:image1:EnableCallbacks_RBV
13SIM1:_image1_min_callback_time 13SIM1:image1:MinCallbackTime_RBV
13SIM1:_image1_nd_array_address 13SIM1:image1:NDArrayAddress_RBV
13SIM1:_image1_nd_array_port 13SIM1:image1:NDArrayPort_RBV
13SIM1:_image1_ndimensions 13SIM1:image1:NDimensions_RBV
13SIM1:_image1_plugin_type 13SIM1:image1:PluginType_RBV
13SIM1:_image1_queue_free 13SIM1:image1:QueueFree
13SIM1:_image1_queue_free_low 13SIM1:image1:QueueFreeLow
13SIM1:_image1_queue_size 13SIM1:image1:QueueSize
13SIM1:_image1_queue_use 13SIM1:image1:QueueUse
13SIM1:_image1_queue_use_high 13SIM1:image1:QueueUseHIGH
13SIM1:_image1_queue_use_hihi 13SIM1:image1:QueueUseHIHI
13SIM1:_image1_time_stamp 13SIM1:image1:TimeStamp_RBV
13SIM1:_image1_unique_id 13SIM1:image1:UniqueId_RBV
13SIM1:_image1_array_data 13SIM1:image1:ArrayData
13SIM1:_hdf1_array_callbacks 13SIM1:HDF1:ArrayCallbacks_RBV
13SIM1:_hdf1_enable_callbacks 13SIM1:HDF1:EnableCallbacks_RBV
13SIM1:_hdf1_auto_increment 13SIM1:HDF1:AutoIncrement_RBV
13SIM1:_hdf1_auto_save 13SIM1:HDF1:AutoSave_RBV
13SIM1:_hdf1_file_path 13SIM1:HDF1:FilePath_RBV
13SIM1:_hdf1_file_name 13SIM1:HDF1:FileName_RBV
13SIM1:_hdf1_file_number 13SIM1:HDF1:FileNumber_RBV
13SIM1:_hdf1_file_template 13SIM1:HDF1:FileTemplate_RBV
13SIM1:_hdf1_file_write_mode 13SIM1:HDF1:FileWriteMode_RBV
13SIM1:_hdf1_full_file_name 13SIM1:HDF1:FullFileName_RBV
13SIM1:_hdf1_xml_layout_file 13SIM1:HDF1:XMLFileName_RBV
MarkRivers commented 7 years ago
/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/epics/ca.py:1248: UserWarning: ca.get('13SIM1:image1:ArrayData') timed out after 1.00 seconds.
  warnings.warn(msg % (name(chid), timeout))

Could that error be because you have not defined EPICS_CA_MAX_ARRAY_BYTES correctly on both the server and client?

prjemian commented 7 years ago

After that output, the remainder of the session produces the ValueError, as above.

Transient Scan ID: 149 Persistent Unique Scan ID: '74ae6602-8af8-4706-a806-376c86e63ce7' /home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/epics/ca.py:1248: UserWarning: ca.get('13SIM1:image1:ArrayData') timed out after 1.00 seconds. warnings.warn(msg % (name(chid), timeout)) /home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/epics/ca.py:1248: UserWarning: ca.get('13SIM1:image1:ArrayData') timed out after 8.08 seconds. warnings.warn(msg % (name(chid), timeout)) generator interlace_tomo_scan ['74ae66'] (scan num: 149) Traceback (most recent call last): File "/home/prjemian/Documents/eclipse/use_bluesky/plans/standlone.py", line 326, in RE(tomo_plan, tomo_callbacks, md=dict(developer=True)) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 599, in __call__ raise exc File "/home/prjemian/Apps/BlueSky/lib/python3.5/asyncio/tasks.py", line 239, in _step result = coro.send(None) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 1007, in _run raise err File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 905, in _run msg = self._plan_stack[-1].send(resp) File "/home/prjemian/Documents/eclipse/use_bluesky/plans/interlace_tomo.py", line 179, in interlace_tomo_scan return (yield from inner_scan()) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 47, in dec_inner return (yield from plan) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1510, in stage_wrapper return (yield from finalize_wrapper(inner(), unstage_devices())) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1025, in finalize_wrapper ret = yield from plan File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1508, in inner return (yield from plan) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 47, in dec_inner return (yield from plan) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 873, in run_wrapper yield from plan File "/home/prjemian/Documents/eclipse/use_bluesky/plans/interlace_tomo.py", line 177, in inner_scan yield from per_step(detectors, motor, projection) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 2010, in one_1d_step return (yield from trigger_and_read(list(detectors) + [motor])) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1819, in trigger_and_read return (yield from rewindable_wrapper(pchain(*plan_stack), rewindable)) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 1222, in rewindable_wrapper return (yield from plan) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/plans.py", line 288, in pchain rets.append((yield from p)) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/utils.py", line 122, in single_gen return (yield msg) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 956, in _run response = yield from coro(msg) File "/home/prjemian/Apps/BlueSky/lib/python3.5/asyncio/coroutines.py", line 206, in coro res = func(*args, **kw) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/bluesky/run_engine.py", line 1269, in _read data_keys = obj.describe() File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/device.py", line 728, in describe res.update(self._describe_attr_list(self.read_attrs)) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/device.py", line 721, in _describe_attr_list desc.update(obj.describe()) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/device.py", line 728, in describe res.update(self._describe_attr_list(self.read_attrs)) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/device.py", line 721, in _describe_attr_list desc.update(obj.describe()) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/signal.py", line 570, in describe desc['dtype'] = data_type(val) File "/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/ophyd/utils/epics_pvs.py", line 464, in data_type raise ValueError('{} not a valid type (int, float, ndarray, str)'.format(val)) ValueError: None not a valid type (int, float, ndarray, str)
prjemian commented 7 years ago

I'm bothered by the warning (from ca.py) about PV 13SIM1:image1:ArrayData

ca.py:1248: UserWarning: ca.get('13SIM1:image1:ArrayData')  timed out after 1.00 seconds.

The script ensures that this is defined (but perhaps this should be defined for sure before ophyd is first imported):

EPICS_CA_MAX_ARRAY_BYTES = "10000000" # also set in SimDetector's IOC (remove the comment there)
os.environ['EPICS_CA_MAX_ARRAY_BYTES'] = EPICS_CA_MAX_ARRAY_BYTES

No. Defining that before any imports (other than import os) did not affect the ValueError or the warning from ca.py.

tacaswell commented 7 years ago

It is coming out of det.describe() which should get you a trace back you can reasonably %debug into when we try to work out what type the PV is (which we probably should do by asking the PV, but have not gotten to fixing that).

You can also look at det.read() to see what values look funny.

prjemian commented 7 years ago

@MarkRivers : Both IOC and client use the same setting for

EPICS_CA_MAX_ARRAY_BYTES=10000000
prjemian commented 7 years ago

with

from pprint import pprint
pprint(simdet.read())

two dtypes in the report have unusual looking values (unsigned of length 34 & 35):

dtype='<U34'
dtype='<U35'
/home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/epics/ca.py:1248: UserWarning: ca.get('13SIM1:image1:ArrayData') timed out after 1.00 seconds. warnings.warn(msg % (name(chid), timeout)) /home/prjemian/Apps/BlueSky/lib/python3.5/site-packages/epics/ca.py:1248: UserWarning: ca.get('13SIM1:image1:ArrayData') timed out after 8.08 seconds. warnings.warn(msg % (name(chid), timeout)) OrderedDict([('13SIM1:_configuration_names', {'timestamp': 1492465863.8643456, 'value': array([], dtype=float64)}), ('13SIM1:_cam_configuration_names', {'timestamp': 1492465863.7257802, 'value': array([], dtype=float64)}), ('13SIM1:_cam_array_counter', {'timestamp': 1492440398.668719, 'value': 0}), ('13SIM1:_cam_array_rate', {'timestamp': 1492465870.15809, 'value': 0.0}), ('13SIM1:_cam_asyn_io', {'timestamp': 631152000.0, 'value': 0}), ('13SIM1:_cam_nd_attributes_file', {'timestamp': 1492440398.65821, 'value': ''}), ('13SIM1:_cam_pool_alloc_buffers', {'timestamp': 1492465870.157292, 'value': 0}), ('13SIM1:_cam_pool_free_buffers', {'timestamp': 1492465870.157301, 'value': 0}), ('13SIM1:_cam_pool_max_buffers', {'timestamp': 1492440398.664065, 'value': 0}), ('13SIM1:_cam_pool_max_mem', {'timestamp': 1492440398.657661, 'value': 0.0}), ('13SIM1:_cam_pool_used_buffers', {'timestamp': 1492465870.157308, 'value': 0.0}), ('13SIM1:_cam_pool_used_mem', {'timestamp': 1492465870.157262, 'value': 0.0}), ('13SIM1:_cam_port_name', {'timestamp': 1492440398.668693, 'value': 'SIM1'}), ('13SIM1:_cam_acquire', {'timestamp': 1492440398.668744, 'value': 0}), ('13SIM1:_cam_acquire_period', {'timestamp': 1492440398.668772, 'value': 0.005}), ('13SIM1:_cam_acquire_time', {'timestamp': 1492440398.668771, 'value': 0.5}), ('13SIM1:_cam_array_callbacks', {'timestamp': 1492440398.668823, 'value': 1}), ('13SIM1:_cam_array_size_array_size_x', {'timestamp': 1492440398.668699, 'value': 1024}), ('13SIM1:_cam_array_size_array_size_z', {'timestamp': 1492440398.668702, 'value': 0}), ('13SIM1:_cam_array_size_array_size_y', {'timestamp': 1492440398.6687, 'value': 1024}), ('13SIM1:_cam_array_size_bytes', {'timestamp': 1492440398.668703, 'value': 0}), ('13SIM1:_cam_bin_x', {'timestamp': 1492440398.66876, 'value': 1}), ('13SIM1:_cam_bin_y', {'timestamp': 1492440398.668762, 'value': 1}), ('13SIM1:_cam_color_mode', {'timestamp': 1492440398.668707, 'value': 0}), ('13SIM1:_cam_data_type', {'timestamp': 1492440398.668706, 'value': 1}), ('13SIM1:_cam_detector_state', {'timestamp': 1492440398.668725, 'value': 0}), ('13SIM1:_cam_frame_type', {'timestamp': 1492440398.668826, 'value': 0}), ('13SIM1:_cam_gain', {'timestamp': 1492440398.668799, 'value': 1.0}), ('13SIM1:_cam_image_mode', {'timestamp': 1492451550.778358, 'value': 0}), ('13SIM1:_cam_manufacturer', {'timestamp': 1492440398.668749, 'value': 'Simulated detector'}), ('13SIM1:_cam_max_size_max_size_y', {'timestamp': 1492440398.668723, 'value': 1024}), ('13SIM1:_cam_max_size_max_size_x', {'timestamp': 1492440398.668721, 'value': 1024}), ('13SIM1:_cam_min_x', {'timestamp': 1492440398.668757, 'value': 0}), ('13SIM1:_cam_min_y', {'timestamp': 1492440398.668759, 'value': 0}), ('13SIM1:_cam_model', {'timestamp': 1492440398.668751, 'value': 'Basic simulator'}), ('13SIM1:_cam_num_exposures', {'timestamp': 1492440398.668834, 'value': 1}), ('13SIM1:_cam_num_exposures_counter', {'timestamp': 1492440398.668729, 'value': 0}), ('13SIM1:_cam_num_images', {'timestamp': 1492440398.668773, 'value': 1}), ('13SIM1:_cam_num_images_counter', {'timestamp': 1492440398.668727, 'value': 0}), ('13SIM1:_cam_read_status', {'timestamp': 631152000.0, 'value': 1}), ('13SIM1:_cam_reverse_reverse_y', {'timestamp': 1492440398.668765, 'value': 0}), ('13SIM1:_cam_reverse_reverse_x', {'timestamp': 1492440398.668763, 'value': 0}), ('13SIM1:_cam_shutter_close_delay', {'timestamp': 1492440398.668802, 'value': 0.0}), ('13SIM1:_cam_shutter_close_epics', {'timestamp': 631152000.0, 'value': 0.0}), ('13SIM1:_cam_shutter_control', {'timestamp': 1492440398.668825, 'value': 0}), ('13SIM1:_cam_shutter_control_epics', {'timestamp': 631152000.0, 'value': 0}), ('13SIM1:_cam_shutter_fanout', {'timestamp': 631152000.0, 'value': 0}), ('13SIM1:_cam_shutter_mode', {'timestamp': 1492440398.668829, 'value': 0}), ('13SIM1:_cam_shutter_open_delay', {'timestamp': 1492440398.6688, 'value': 0.0}), ('13SIM1:_cam_shutter_open_epics', {'timestamp': 631152000.0, 'value': 0.0}), ('13SIM1:_cam_shutter_status_epics', {'timestamp': 631152000.0, 'value': 0}), ('13SIM1:_cam_shutter_status', {'timestamp': 1492440398.668742, 'value': 0}), ('13SIM1:_cam_size_size_x', {'timestamp': 1492440398.668766, 'value': 40}), ('13SIM1:_cam_size_size_y', {'timestamp': 1492440398.668767, 'value': 30}), ('13SIM1:_cam_status_message', {'timestamp': 1492440398.668745, 'value': ''}), ('13SIM1:_cam_string_from_server', {'timestamp': 1492440398.668748, 'value': ''}), ('13SIM1:_cam_string_to_server', {'timestamp': 1492440398.668747, 'value': ''}), ('13SIM1:_cam_temperature', {'timestamp': 1492440398.668803, 'value': 25.0}), ('13SIM1:_cam_temperature_actual', {'timestamp': 631152000.0, 'value': 0.0}), ('13SIM1:_cam_time_remaining', {'timestamp': 1492440398.668731, 'value': 0.0}), ('13SIM1:_cam_trigger_mode', {'timestamp': 1492440398.668828, 'value': 0}), ('13SIM1:_cam_gain_rgb_gain_red', {'timestamp': 1492440398.668781, 'value': 1.0}), ('13SIM1:_cam_gain_rgb_gain_green', {'timestamp': 1492440398.668782, 'value': 1.0}), ('13SIM1:_cam_gain_rgb_gain_blue', {'timestamp': 1492440398.668784, 'value': 1.0}), ('13SIM1:_cam_gain_xy_gain_y', {'timestamp': 1492440398.668779, 'value': 1.0}), ('13SIM1:_cam_gain_xy_gain_x', {'timestamp': 1492440398.668778, 'value': 1.0}), ('13SIM1:_cam_noise', {'timestamp': 1492440398.668775, 'value': 3}), ('13SIM1:_cam_peak_num_peak_num_x', {'timestamp': 1492440398.668792, 'value': 1}), ('13SIM1:_cam_peak_num_peak_num_y', {'timestamp': 1492440398.668793, 'value': 1}), ('13SIM1:_cam_peak_start_peak_start_y', {'timestamp': 1492440398.668788, 'value': 1}), ('13SIM1:_cam_peak_start_peak_start_x', {'timestamp': 1492440398.668787, 'value': 1}), ('13SIM1:_cam_peak_step_peak_step_y', {'timestamp': 1492440398.668796, 'value': 1}), ('13SIM1:_cam_peak_step_peak_step_x', {'timestamp': 1492440398.668795, 'value': 1}), ('13SIM1:_cam_peak_variation', {'timestamp': 1492440398.668798, 'value': 3}), ('13SIM1:_cam_peak_width_peak_width_x', {'timestamp': 1492440398.668789, 'value': 10}), ('13SIM1:_cam_peak_width_peak_width_y', {'timestamp': 1492440398.668791, 'value': 20}), ('13SIM1:_cam_reset', {'timestamp': 1492440398.668777, 'value': 1}), ('13SIM1:_cam_sim_mode', {'timestamp': 1492440398.668785, 'value': 0}), ('13SIM1:_image1_configuration_names', {'timestamp': 1492465863.8657985, 'value': array(['13SIM1:_image1_port_name', '13SIM1:_image1_nd_array_port', '13SIM1:_image1_enable', '13SIM1:_image1_blocking_callbacks', '13SIM1:_image1_plugin_type', '13SIM1:_image1_asyn_pipeline_config', '13SIM1:_image1_configuration_names'], dtype='
tacaswell commented 7 years ago

dtype='<U34' dtype='<U35'

These are little endian unicode of fixed length, iirc unsinged is 'u'

prjemian commented 7 years ago

something is out of place with my setup of the code above, and that causes the problems reported, reverting back to a known working case and continuing from there with these lessons learned:

tacaswell commented 7 years ago

One other thing that came to mind, for area detectors on the floor at NSLS-II we are very selective about what we put into read_attrs, there may in fact be bugs that we just are not seeing.