prjemian commented 6 years ago

We have opted to defer this issue to next milestone.

prjemian commented 6 years ago

@decarlof FYI

decarlof commented 6 years ago

you need to set a PV: ioc_prefix + cam1:FrameType

0 => projections 1 => dark 2 => white

this sets in the detector attribute.xml file the "SaveDest" variable that is then passed to the

detector layout.xml to set "detector_data_destination"

I think this last one is hard coded in the hdf plug in to set the correct label under \exchange: \exchange\ data \exchange\data_dark \exchange\data_white

@aglowacki is the one who implemented and he might add more details

prjemian commented 6 years ago

@decarlof , @aglowacki During testing to develop this support today, found an issue to be resolved in ophyd. See https://github.com/NSLS-II/bluesky/issues/1055

prjemian commented 6 years ago

simple test plan

def darks_flats_images(det, shutter, stage, pos_in, pos_out, n_darks=3, n_flats=4, n_images=5, count_time=0.2, md=None):
    det.cam.stage_sigs["acquire_time"] = count_time

    yield from bps.mv(
        shutter, "close", 
        det.cam.frame_type, 1,   # Background
        det.hdf1.num_capture, n_darks + n_flats + n_images,
        )
    yield from bp.count([det], num=n_darks, md=md)

    yield from bps.mv(
        shutter, "open", 
        stage, pos_out,
        det.cam.frame_type, 2)   # FlatField
    yield from bp.count([det], num=n_flats, md=md)

    yield from bps.mv(
        shutter, "open", 
        stage, pos_in,
        det.cam.frame_type, 0)   # images
    yield from bp.count([det], num=n_images, md=md)

    yield from bps.mv(shutter, "close")

executed in Bluesky

In [4]: RE(darks_flats_images(adsimdet, shutter, m1, 0, 1))
Transient Scan ID: 202     Time: 2018/06/15 17:32:40
Persistent Unique Scan ID: '31ccc2d9-e56d-4631-9bb6-fc96d8bb2980'
New stream: 'aps_sr_current_monitor'
New stream: 'primary'                                                                                                                                                                                
+-----------+------------+
|   seq_num |       time |
+-----------+------------+
|         1 | 17:32:43.0 |
|         2 | 17:32:43.4 |                                                                                                                                                                           
|         3 | 17:32:43.6 |                                                                                                                                                                           
+-----------+------------+
generator count ['31ccc2d9'] (scan num: 202)

Transient Scan ID: 203     Time: 2018/06/15 17:32:47                                                                                                                                                 
Persistent Unique Scan ID: '9d045f73-4cbc-4725-bdfe-5ebebcbbd641'                                                                                                                                    
New stream: 'aps_sr_current_monitor'
New stream: 'primary'                                                                                                                                                                                
+-----------+------------+
|   seq_num |       time |
+-----------+------------+
|         1 | 17:32:47.4 |
|         2 | 17:32:47.7 |                                                                                                                                                                           
|         3 | 17:32:47.9 |                                                                                                                                                                           
|         4 | 17:32:48.1 |                                                                                                                                                                           
+-----------+------------+
generator count ['9d045f73'] (scan num: 203)

Transient Scan ID: 204     Time: 2018/06/15 17:32:48
Persistent Unique Scan ID: 'a473dcd6-86fb-4118-ba0b-71fb86621124'
New stream: 'aps_sr_current_monitor'
New stream: 'primary'                                                                                                                                                                                
+-----------+------------+
|   seq_num |       time |
+-----------+------------+
|         1 | 17:32:48.9 |
|         2 | 17:32:49.7 |                                                                                                                                                                           
|         3 | 17:32:50.0 |                                                                                                                                                                           
|         4 | 17:32:50.3 |                                                                                                                                                                           
|         5 | 17:32:50.5 |                                                                                                                                                                           
+-----------+------------+
generator count ['a473dcd6'] (scan num: 204)

Out[4]:                                                                                                                                                                                              
('31ccc2d9-e56d-4631-9bb6-fc96d8bb2980',
 '9d045f73-4cbc-4725-bdfe-5ebebcbbd641',
 'a473dcd6-86fb-4118-ba0b-71fb86621124')

HDF5 file data location

puts the data in the right place of the HDF5 file

jemian@otz .../06/15 $ for f in `ls *.h5`; do punx st $f | head -20; done
/tmp/simdet/2018/06/15/22f4d049-180d-4206-82fe_000000.h5
  detector
    darks:uint8[1,1024,1024]
      @NDArrayDimBinning = 1
      @NDArrayDimOffset = 0
      @NDArrayDimReverse = 0
      @NDArrayNumDims = 2
      @description = DarkData
    flats:uint8[4,1024,1024]
      @NDArrayDimBinning = 1
      @NDArrayDimOffset = 0
      @NDArrayDimReverse = 0
      @NDArrayNumDims = 2
      @description = WhiteData
    images:uint8[1,1024,1024]
      @NDArrayDimBinning = 1
      @NDArrayDimOffset = 0
      @NDArrayDimReverse = 0
      @NDArrayNumDims = 2
      @description = ImageData
/tmp/simdet/2018/06/15/394675ce-7156-4eca-acb1_000000.h5
  detector
    darks:uint8[3,1024,1024]
      @NDArrayDimBinning = 1
      @NDArrayDimOffset = 0
      @NDArrayDimReverse = 0
      @NDArrayNumDims = 2
      @description = DarkData
    flats:uint8[1,1024,1024]
      @NDArrayDimBinning = 1
      @NDArrayDimOffset = 0
      @NDArrayDimReverse = 0
      @NDArrayNumDims = 2
      @description = WhiteData
    images:uint8[1,1024,1024]
      @NDArrayDimBinning = 1
      @NDArrayDimOffset = 0
      @NDArrayDimReverse = 0
      @NDArrayNumDims = 2
      @description = ImageData
/tmp/simdet/2018/06/15/f14a0c4d-063f-43ad-b31b_000000.h5
  detector
    darks:uint8[1,1024,1024]
      @NDArrayDimBinning = 1
      @NDArrayDimOffset = 0
      @NDArrayDimReverse = 0
      @NDArrayNumDims = 2
      @description = DarkData
    flats:uint8[1,1024,1024]
      @NDArrayDimBinning = 1
      @NDArrayDimOffset = 0
      @NDArrayDimReverse = 0
      @NDArrayNumDims = 2
      @description = WhiteData
    images:uint8[5,1024,1024]
      @NDArrayDimBinning = 1
      @NDArrayDimOffset = 0
      @NDArrayDimReverse = 0
      @NDArrayNumDims = 2
      @description = ImageData

but Bluesky creates one file for images, another for flats, and another for darks. Gotta change that so all go in one file.

attributes XML

<Attributes
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://epics.aps.anl.gov/areaDetector/attributes ../attributes.xsd"
    >

    <Attribute name="DateTimeStart" type="EPICS_PV" source="S:IOC:timeOfDayForm1SI"  dbrtype="DBR_STRING"/>
    <Attribute name="DateTimeEnd"   type="EPICS_PV" source="S:IOC:timeOfDayForm1SI"  dbrtype="DBR_STRING"/>
    <Attribute name="Current"       type="EPICS_PV" source="S:SRcurrentAI"           dbrtype="DBR_DOUBLE"/>
    <Attribute name="TopUpStatus"   type="EPICS_PV" source="S:TopUpStatus"           dbrtype="DBR_ENUM"/>
    <Attribute name="BeamMode"      type="EPICS_PV" source="OPS:message3"            dbrtype="DBR_STRING"/>

    <Attribute name="SaveDest"      type="EPICS_PV" source="otzSIM1:cam1:FrameType"  dbrtype="DBR_STRING"/>

</Attributes>

layout XML

<hdf5_layout>
  <global name="detector_data_destination" ndattribute="SaveDest" />
  <group name="detector">
    <dataset name="images" source="detector" det_default="true">
        <attribute name="description" source="constant" value="ImageData" type="string" />
    </dataset>
    <dataset name="flats" source="detector" det_default="true">
        <attribute name="description" source="constant" value="WhiteData" type="string" />
    </dataset>
    <dataset name="darks" source="detector" det_default="true">
        <attribute name="description" source="constant" value="DarkData" type="string" />
    </dataset>
    <group name="metadata" ndattr_default="true" />
  </group>
</hdf5_layout>

prjemian commented 6 years ago

refer to this jupyter notebook for a demo: https://github.com/BCDA-APS/use_bluesky/blob/master/notebooks/images_darks_flats.ipynb

prjemian commented 6 years ago

Tekin wants the attribute name that tells if flat, image, or dark

prjemian commented 6 years ago

Attribute name is SaveDest (according to the notebook cited above).

prjemian commented 6 years ago

Current settings

[user2bmb@lyra,43,~]$ caget 2bmbPG3:cam1:FrameType
2bmbPG3:cam1:FrameType         Normal
[user2bmb@lyra,44,~]$ caget 2bmbPG3:cam1:FrameType.ZRST
2bmbPG3:cam1:FrameType.ZRST    Normal
[user2bmb@lyra,45,~]$ caget 2bmbPG3:cam1:FrameType.ONST
2bmbPG3:cam1:FrameType.ONST    Background
[user2bmb@lyra,46,~]$ caget 2bmbPG3:cam1:FrameType.TWST
2bmbPG3:cam1:FrameType.TWST    FlatField

prjemian commented 6 years ago

As stated in the cited notebook:

The HDF5 addresses shown are for the Data Exchange format. Both these PVs (2bmbPG3:cam1:FrameType and 2bmbPG3:cam1:FrameType_RBV) should be configured so the values on the RHS are the HDF5 address in the file where the image frame should be stored:

PV.ZRST = /exchange/data
PV.ONST = /exchange/data_dark
PV.TWST = /exchange/data_white
# make sure to put these in autosave so they are restored when the IOC is restarted!

prjemian commented 6 years ago

Important Points

algorithm to get darks, whites, and images in one HDF5 file:

cam is in Multiple mode
hdf1 is in Stream mode with Capture on
hdf1 NumCapture is total number of frames (darks+whites+images)
compression is optional, tends to slow down tomo, good for developer use
detector.stage() opens the HDF5 file
shoot all images between stage and unstage
for each frame type
- set NumImages on cam
- cam Acquire
- wait until hdf1 NumCaptured reaches expected cumulative number of frames
detector.unstage() closes the HDF5 file

APS-2BM-MIC / ipython-user2bmb

add darks & flats to current tomo_scan() plan #27

simple test plan

executed in Bluesky

HDF5 file data location

attributes XML

layout XML

Important Points