refactor DM support so it can be called from bluesky or pyepics

prjemian commented 5 years ago

Transitional support for use in bluesky and SPEC, refactor the DM support for use in either bluesky or pyepics. Will need PyEpics structure similar to what is in 35-DM-support.py that uses epics.PV instead of EpicsSignal.

[x] build data structure
[x] write the workflow file
[x] execute the workflow from SPEC
[x] execute the workflow from bluesky

sureshnaps commented 4 years ago

@prjemian if possible, i would like to request a higher priority for this

prjemian commented 4 years ago

This can start as soon as #54 is merged. There's content in that PR that overlaps this issue so best to do them in sequence.

prjemian commented 4 years ago

SPEC will need this code to run as a separate background process and trigger jobs with some EPICS PVs (not yet identified or created).

SPEC must write the metadata PVs (already does this)
SPEC needs to pass a couple globals (xpcs_qmap_file, ...) into PVs
PV to trigger a job (SPEC sets this to 1, python sets to zero when job is done)
- python reads metadata PVs and writes HDF5 workflow file
- python kicks off appropriate transfer or analysis workflow

sureshnaps commented 4 years ago

So the commented ones that uses bluesky params need to be changed to epics pvs, seems like that is it. I will do that during our work tomorrow. Rest looks good.

qzhang234 commented 4 years ago

Here is the error message from Bluesky upon start-up at 11:20 pm, 09/10. Copied here per @sureshnaps 's request. Hope it's helpful, @prjemian

Error_Log.txt

prjemian commented 4 years ago

@sureshnaps - There is more work on this PR before it is ready. Stay tuned ...

qzhang234 commented 4 years ago

Pete, Suresh:

I'm running 15 to 20 min late. Please start without me.

Very sorry about that,

QZ

From: Pete R Jemian notifications@github.com Sent: Wednesday, September 11, 2019 7:52 AM To: aps-8id-trr/ipython-8idiuser ipython-8idiuser@noreply.github.com Cc: Zhang, Qingteng qzhang234@anl.gov; Comment comment@noreply.github.com Subject: Re: [aps-8id-trr/ipython-8idiuser] refactor DM support so it can be called from bluesky or pyepics (#36)

@sureshnapshttps://github.com/sureshnaps - There is more work on this PR before it is ready. Stay tuned ...

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/aps-8id-trr/ipython-8idiuser/issues/36?email_source=notifications&email_token=ALPJBQRZDM6M6QC6T3C27ULQJDZ3DA5CNFSM4IQEEG22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6OR5EI#issuecomment-530390673, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALPJBQTUIWJGJ4ZEGC7NQQLQJDZ3DANCNFSM4IQEEG2Q.

prjemian commented 4 years ago

starting over on the branch, once we get some new metadata PVs (#71)

prjemian commented 4 years ago

As a python program, this connects and prints in a table in under one second:

(bluesky) bash-4.2$ date; python ./spec_DM_support.py ; date
Fri Sep 13 16:47:39 CDT 2019
====== ======================================= ============= ======================== =======================================
type   name                                    PV            description              value
====== ======================================= ============= ======================== =======================================
number hdf_metadata_version                    8idi:Reg1     batchinfo_ver            15.0
number detNum                                  8idi:Reg2     detector PI8,DAL5,SMD6   25.0
number geometry_num                            8idi:Reg3     geometry (0-Trans,1-Ref) 0.0
number detector_distance                       8idi:Reg5     rr (Sample-Det. Dist.)   3930.0
number compression                             8idi:Reg8     compression              1.0
number source_begin_beam_intensity_incident    8idi:Reg9     beam_i_vacuum            1.0
number source_begin_beam_intensity_transmitted 8idi:Reg10    beam_i                   573112333.64046
number beam_center_x                           8idi:Reg11    x0 (beam0-X)             674.90852553542
number beam_center_y                           8idi:Reg12    y0 (beam0-Y)             157.15317133443
number stage_zero_x                            8idi:Reg13    ccdx0 (ccdx at beam0)    209.0
number stage_zero_z                            8idi:Reg14    ccdz0 (ccdz at beam0)    31.5
number xspec                                   8idi:Reg15    xspec (H pos for spec)   876.71777020394
number zspec                                   8idi:Reg16    yspec (V pos for spec)   148.45656772884
number ccdzspec                                8idi:Reg17    ccdxspec (ccdx at xspec) 250.0
number ccdxspec                                8idi:Reg18    ccdzspec (ccdz at zspec) 72.0
number angle                                   8idi:Reg19    nominal_angle            0.2
number roi_x1                                  8idi:Reg101   col_begin                0.0
number roi_x2                                  8idi:Reg102   col_end                  1555.0
number roi_y1                                  8idi:Reg103   row_begin                0.0
number roi_y2                                  8idi:Reg104   row_end                  515.0
number cols                                    8idi:Reg105   cols                     1556.0
number rows                                    8idi:Reg106   rows                     516.0
number kinetics_state                          8idi:Reg107   kinetics                 0.0
number kinetics_window_size                    8idi:Reg108   kinwinsize               0.0
number kinetics_top                            8idi:Reg109   slice top                0.0
number attenuation                             8idi:Reg110   attenuation              737354.77126117
number dark_begin                              8idi:Reg111   ndark0                   -1.0
number dark_end                                8idi:Reg112   ndarkend                 -1.0
number data_begin                              8idi:Reg113   ndata0                   1.0
number data_end                                8idi:Reg114   ndataend                 10.0
number exposure_time                           8idi:Reg115   exposure_time            1.0
number exposure_period                         8idi:Reg116   exposure_period          2.0
number specscan_dark_number                    8idi:Reg117   dark scan_N              -1.0
number specscan_data_number                    8idi:Reg118   data scan_N              680.0
number stage_x                                 8idi:Reg119   ccdx                     365.0
number stage_z                                 8idi:Reg120   ccdz                     -200.0
number source_begin_current                    8idi:Reg121   ring_i_beg               0.201296328604
number source_end_current                      8idi:Reg122   ring_i_end               0.201296328604
number I0mon                                   8idi:Reg123   I0Mon                    24.0
number burst_mode_state                        8idi:Reg124   burst_enabled            0.0
number number_of_bursts                        8idi:Reg125   number_of_bursts         0.0
number first_usable_burst                      8idi:Reg126   first_usable_burst       0.0
number last_usable_burst                       8idi:Reg127   last_usable_burst        0.0
number beam_size_H                             8idi:Reg151   beam_size_H              60.0013
number beam_size_V                             8idi:Reg152   beam_size_V              59.9952
number source_begin_energy                     8idi:Reg153   source_begin_energy      7.495
number temperature_A                           8idi:Reg154   temperature_A            -273.1499938964844
number temperature_B                           8idi:Reg155   temperature_B            -273.1499938964844
number temperature_A_set                       8idi:Reg156   temperature_A_set        25.0
number temperature_B_set                       8idi:Reg157   temperature_B_set        25.0
number translation_x                           8idi:Reg158   translation_x            0.0
number translation_y                           8idi:Reg159   translation_y            0.0
number translation_z                           8idi:Reg160   translation_z            0.0
number translation_table_x                     8idi:Reg161   translation_table_x      0.0
number translation_table_y                     8idi:Reg162   translation_table_y      0.0
number translation_table_z                     8idi:Reg163   translation_table_z      10.000070000000001
number sample_pitch                            8idi:Reg164   sample_pitch             0.0
number sample_roll                             8idi:Reg165   sample_roll              0.0
number sample_yaw                              8idi:Reg166   sample_yaw               0.0
number pid1_set                                8idi:Reg167   pid1_set                 0.0
number pid2_set                                8idi:Reg168   pid2_set                 0.0
number scan_id                                 8idi:Reg169   bluesky_scan_number      32
text   specfile                                8idi:StrReg1  specfile                 weichen20190813
text   root_folder                             8idi:StrReg2  root folder              /home/8-id-i/2019-2/jemian_201908/A024/
text   user_data_folder                        8idi:StrReg3  user_data_folder         /home/8-id-i/2019-2/jemian_201908/A024
text   data_folder                             8idi:StrReg4  data_folder              A024
text   datafilename                            8idi:StrReg5  datafilename             A022_00001-00010.imm
text   source_begin_datetime                   8idi:StrReg6  start_time               Wed 11 Sep 2019 10:56:12 AM
text   source_end_datetime                     8idi:StrReg7  end_time                 Wed 11 Sep 2019 10:56:22 AM
text   data_subfolder                          8idi:StrReg10 data_subfolder           A186_DOHE04_Yb010_att0_Uq0_00150
text   uid                                     8idi:StrReg11 bluesky_scan_uid         35e39c78-46b6-495f-9aa9-df680a2bc860
====== ======================================= ============= ======================== =======================================

Fri Sep 13 16:47:39 CDT 2019

sureshnaps commented 4 years ago

that is brillant @prjemian

prjemian commented 4 years ago

not happy with tests yet

connected in 1.679s
finished in 1.736s
{5: 'DALSA', 8: 'PI Princeton Instruments', 13: 'PI Princeton Instruments', 20: 'ANL-LBL FastCCD Detector', 25: 'LAMBDA', 30: 'EIGER', 35: 'UFXC_128x256', 45: 'RIGAKU500K', 46: 'RIGAKU500K_NoGap'}
before writing HDF5: /tmp/workflow.h5 exists: False
after writing HDF5: /tmp/workflow.h5 exists: False

Why no file written and no exception raised?

prjemian commented 4 years ago

It's that as_bluesky_plan test. Something is off with that. Remove that argument. Means that ad_acquire.AD_Acquire() will have to change, as well. (no yield from ... there)

prjemian commented 4 years ago

now

connected in 3.391s
finished in 3.499s
{5: 'DALSA', 8: 'PI Princeton Instruments', 13: 'PI Princeton Instruments', 20: 'ANL-LBL FastCCD Detector', 25: 'LAMBDA', 30: 'EIGER', 35: 'UFXC_128x256', 45: 'RIGAKU500K', 46: 'RIGAKU500K_NoGap'}
before writing HDF5: /tmp/workflow.h5 exists: False
after writing HDF5: /tmp/workflow.h5 exists: True

prjemian commented 4 years ago

Here's the HDF5 workflow file (uses same code as bluesky uses to write this file):

(bluesky) bash-4.2$ punx tree /tmp/workflow.h5

!!! WARNING: this program is not ready for distribution.

/home/beams/8IDIUSER/.conda/envs/bluesky/lib/python3.7/site-packages/h5py/_hl/dataset.py:313: H5pyDeprecationWarning: dataset.value has been deprecated. Use dataset[()] instead.
  "Use dataset[()] instead.", H5pyDeprecationWarning)
/tmp/workflow.h5
  hdf_metadata_version:float64[1,1] = __array
    __array = [
        [15.0]
      ]
  measurement
    instrument
      acquisition
        angle:float64[1,1] = __array
          __array = [
              [-1.0]
            ]
        attenuation:float64[1,1] = __array
          __array = [
              [737354.77126117]
            ]
        beam_center_x:float64[1,1] = __array
          __array = [
              [674.90852553542]
            ]
        beam_center_y:float64[1,1] = __array
          __array = [
              [157.15317133443]
            ]
        beam_size_H:float64[1,1] = __array
          __array = [
              [60.0013]
            ]
        beam_size_V:float64[1,1] = __array
          __array = [
              [59.9952]
            ]
        ccdxspec:float64[1,1] = __array
          __array = [
              [-1.0]
            ]
        ccdzspec:float64[1,1] = __array
          __array = [
              [-1.0]
            ]
        compression:CHAR = ENABLED
        dark_begin:uint64[1,1] = __array
          __array = [
              [0]
            ]
        dark_end:uint64[1,1] = __array
          __array = [
              [0]
            ]
        data_begin:uint64[1,1] = __array
          __array = [
              [1]
            ]
        data_end:uint64[1,1] = __array
          __array = [
              [10]
            ]
        data_folder:CHAR = A024
        datafilename:CHAR = A022_00001-00010.imm
        parent_folder:CHAR = jemian_201908
        root_folder:CHAR = /home/8-id-i/2019-2/jemian_201908/A024/A186_DOHE04_Yb010_att0_Uq0_00150/
        specfile:CHAR = weichen20190813
        specscan_dark_number:uint64[1,1] = __array
          __array = [
              [0]
            ]
        specscan_data_number:uint64[1,1] = __array
          __array = [
              [680]
            ]
        stage_x:float64[1,1] = __array
          __array = [
              [365.0]
            ]
        stage_z:float64[1,1] = __array
          __array = [
              [-200.0]
            ]
        stage_zero_x:float64[1,1] = __array
          __array = [
              [209.0]
            ]
        stage_zero_z:float64[1,1] = __array
          __array = [
              [31.5]
            ]
        xspec:float64[1,1] = __array
          __array = [
              [-1.0]
            ]
        zspec:float64[1,1] = __array
          __array = [
              [-1.0]
            ]
      detector
        adu_per_photon:float64[1,1] = __array
          __array = [
              [1.0]
            ]
        bit_depth:uint32[1,1] = __array
          __array = [
              [12]
            ]
        blemish_enabled:CHAR = ENABLED
        burst_enabled:CHAR = DISABLED
        distance:float64[1,1] = __array
          __array = [
              [3930.0]
            ]
        efficiency:float64[1,1] = __array
          __array = [
              [1.0]
            ]
        exposure_period:float64[1,1] = __array
          __array = [
              [2.0]
            ]
        exposure_time:float64[1,1] = __array
          __array = [
              [1.0]
            ]
        flatfield_enabled:CHAR = ENABLED
        gain:uint32[1,1] = __array
          __array = [
              [1]
            ]
        geometry:CHAR = TRANSMISSION
        kinetics_enabled:CHAR = DISABLED
        lld:float64[1,1] = __array
          __array = [
              [0.0]
            ]
        manufacturer:CHAR = LAMBDA
        sigma:float64[1,1] = __array
          __array = [
              [0.0]
            ]
        x_binning:uint32[1,1] = __array
          __array = [
              [1]
            ]
        x_dimension:uint32[1,1] = __array
          __array = [
              [1556]
            ]
        x_pixel_size:float64[1,1] = __array
          __array = [
              [0.055]
            ]
        y_binning:uint32[1,1] = __array
          __array = [
              [1]
            ]
        y_dimension:uint32[1,1] = __array
          __array = [
              [516]
            ]
        y_pixel_size:float64[1,1] = __array
          __array = [
              [0.055]
            ]
        burst
          first_usable_burst:uint32[1,1] = __array
            __array = [
                [0]
              ]
          last_usable_burst:uint32[1,1] = __array
            __array = [
                [0]
              ]
          number_of_bursts:uint32[1,1] = __array
            __array = [
                [0]
              ]
        kinetics
          first_usable_window:uint32[1,1] = __array
            __array = [
                [0]
              ]
          last_usable_window:uint32[1,1] = __array
            __array = [
                [0]
              ]
          top:uint32[1,1] = __array
            __array = [
                [0]
              ]
          window_size:uint32[1,1] = __array
            __array = [
                [0]
              ]
        roi
          x1:uint32[1,1] = __array
            __array = [
                [0]
              ]
          x2:uint32[1,1] = __array
            __array = [
                [1555]
              ]
          y1:uint32[1,1] = __array
            __array = [
                [0]
              ]
          y2:uint32[1,1] = __array
            __array = [
                [515]
              ]
      source_begin
        beam_intensity_incident:float64[1,1] = __array
          __array = [
              [1.0]
            ]
        beam_intensity_transmitted:float64[1,1] = __array
          __array = [
              [573112333.64046]
            ]
        current:float64[1,1] = __array
          __array = [
              [0.201296328604]
            ]
        datetime:CHAR = Wed 11 Sep 2019 10:56:12 AM
        energy:float64[1,1] = __array
          __array = [
              [7.495]
            ]
      source_end
        current:float64[1,1] = __array
          __array = [
              [0.201296328604]
            ]
        datetime:CHAR = Wed 11 Sep 2019 10:56:22 AM
    sample
      orientation:float64[1,3] = __array
        __array = [
            [0.0, 0.0, 0.0]
          ]
      temperature_A:float64[1,1] = __array
        __array = [
            [-273.1499938964844]
          ]
      temperature_A_set:float64[1,1] = __array
        __array = [
            [25.0]
          ]
      temperature_B:float64[1,1] = __array
        __array = [
            [-273.1499938964844]
          ]
      temperature_B_set:float64[1,1] = __array
        __array = [
            [25.0]
          ]
      thickness:float64[1,1] = __array
        __array = [
            [1.0]
          ]
      translation:float64[1,3] = __array
        __array = [
            [0.0, 0.0, 0.0]
          ]
      translation_table:float64[1,3] = __array
        __array = [
            [0.0, 0.0, 10.000070000000001]
          ]

prjemian commented 4 years ago

Next, need to write code that executes the workflow. Basically an adaptation of tester.py that can be called from command line (as if by SPEC) or from python. Look for similar code in user.ad_acquire.AD_Acquire().

prjemian commented 4 years ago

execute the workflow

For this code to run in both bluesky and SPEC, it cannot rely on common code from apstools, which requires imports of bluesky. Need to copy these functions into APS_DM_8ID.py:

apstools.utils.unix()
apstools.utils.run_in_thread()

Then, the post-acquisition workflow steps (from ad_acquire.py) should also move into APS_DM_8ID.py.

prjemian commented 4 years ago

Now, the entire workflow (make the HDF5 file name, make the HDF5 file, launch the analysis/transfer job in a thread) are all centralized in APS_DM_8IDI.DM_Workflow().start_workflow(): https://github.com/aps-8id-trr/ipython-8idiuser/blob/ba773843f4dfec18547ccb321a6cde8c7892f208/profile_bluesky/startup/spec_support/APS_DM_8IDI.py#L183-L199

This code returns after the unix command is sent (in a separate thread). The thread will run until the unix command returns but the caller (SPEC or bluesky) can continue in parallel to the unix job.

sureshnaps commented 4 years ago

@prjemian this is wonderful. regarding the time it takes, i was confused between "time to connect" and "the other time". The former was 2-3 sec and the difference was very short. Does it mean that the connection still takes 2-3 sec?

prjemian commented 4 years ago

Because the initial connect time can be slow (2-3 seconds), the python code should run as a background process, waiting for SPEC to tell it when to process a workflow. That covers the initial connection problem. Subsequent read/write is very fast, as shown.

@sureshnaps , @qzhang234 : To support SPEC, the code in APS_DM_8IDI.py and spec_DM_support.py needs to listen for SPEC to tell it to call start_workflow().

The registers have all the information needed to start the workflow but need an additional number register to tell it when to start. The caller sets the PV to non-zero (such as 1) to signal the python code to call start_workflow(). When that call returns, the python code sets the PV back to 0. The caller waits for the PV to return to 0 before continuing.

prjemian commented 4 years ago

Perhaps we take three registers:

register	name	description
8idi:Reg170	workflow_start	non-zero: start workflow, 0: workflow started
8idi:Reg171	workflow_helper_alive	increments at 10 Hz (modulo 1e5) when workflow helper is running
8idi:StrReg12	workflow_caller	who set start_workflow (SPEC or bluesky) - avoids contention

The workflow_helper is the background Python process described above.

prjemian commented 4 years ago

Also, I'm concerned that SPEC will take too long just to write the metadata PV registers. If that is the case, we can consider writing metadata to a JSON file (named in another StrReg), then the workflow_helper can read that file when it gets the trigger, write the metadata PVs (should be very fast), then proceed as planned now.

sureshnaps commented 4 years ago

Because the initial connect time can be slow (2-3 seconds), the python code should run as a background process, waiting for SPEC to tell it when to process a workflow.

@sureshnaps , @qzhang234 : To support SPEC, the code in APS_DM_8IDI.py and spec_DM_support.py needs to listen for SPEC to tell it to call start_workflow().

The registers have all the information needed to start the workflow but need an additional number register to tell it when to start. The caller sets the PV to non-zero (such as 1) to signal the python code to call start_workflow(). When that call returns, the python code sets the PV back to 0. The caller waits for the PV to return to 0 before continuing.

@prjemian this is good. May be I am misunderstanding something about this connect time. Let us say, if we do repeated acq's each lasting precisely 2 sec which is pretty close now with Rigaku and some times with Lambda, then is it possible that PVs from acq N are overwritten by the PVs from acq N+1 before the hdf is written?

prjemian commented 4 years ago

Additional rule for setting 8idi:Reg170, 8idi:StrReg12 must be empty string or ready indicating caller can use the workflow helper.

prjemian commented 4 years ago

Regarding acquisition with Rigaku in 2-3 s/measurement, I expect that reading and writing of metadata PVs will be in a couple 100 ms. Since the workflow helper is running in the background and has already connected to the PVs (that's the step that took a few seconds but was only needed once), the only work is to read/write the PVs. Reading is the faster of the two since it involves getting the value from a Python cache.

sureshnaps commented 4 years ago

i am sure spec writes registers fast, not too many of them. if json is not hard, then it is a safer step i think. that is what i was thinking before this.

prjemian commented 4 years ago

To answer your question directly, no. Acquisition frequencies as high as once every second should not be a problem.

On Sat, Sep 14, 2019, 11:57 AM Suresh N. notifications@github.com wrote:

Because the initial connect time can be slow (2-3 seconds), the python code should run as a background process, waiting for SPEC to tell it when to process a workflow.

@sureshnaps https://github.com/sureshnaps , @qzhang234 https://github.com/qzhang234 : To support SPEC, the code in APS_DM_8IDI.py and spec_DM_support.py needs to listen for SPEC to tell it to call start_workflow().

The registers have all the information needed to start the workflow but need an additional number register to tell it when to start. The caller sets the PV to non-zero (such as 1) to signal the python code to call start_workflow(). When that call returns, the python code sets the PV back to 0. The caller waits for the PV to return to 0 before continuing.

@prjemian https://github.com/prjemian this is good. May be I am misunderstanding something about this connect time. Let us say, if we do repeated acq's each lasting precisely 2 sec which is pretty close now with Rigaku and some times with Lambda, then is it possible that PVs from acq N are overwritten by the PVs from acq N+1 before the hdf is written?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aps-8id-trr/ipython-8idiuser/issues/36?email_source=notifications&email_token=AARMUMFWOPO7JSQRHPOR6K3QJUJWNA5CNFSM4IQEEG22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6W7VEY#issuecomment-531495571, or mute the thread https://github.com/notifications/unsubscribe-auth/AARMUMFGAHFZ5NCDTBEUCMLQJUJWNANCNFSM4IQEEG2Q .

qzhang234 commented 4 years ago

So it's the connection to the string registers that takes a couple of seconds and reading/writing to string register should take a fraction of a second, right?

prjemian commented 4 years ago

Right

On Sat, Sep 14, 2019, 2:55 PM qzhang234 notifications@github.com wrote:

So it's the connection to the string registers that takes a couple of seconds and reading/writing to string register should take a fraction of a second, right?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aps-8id-trr/ipython-8idiuser/issues/36?email_source=notifications&email_token=AARMUMGZROM27PDNKI2P2QTQJU6UXA5CNFSM4IQEEG22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6XC4EY#issuecomment-531508755, or mute the thread https://github.com/notifications/unsubscribe-auth/AARMUMHYH6WUYXVQZMU7H2TQJU6UXANCNFSM4IQEEG2Q .

qzhang234 commented 4 years ago

@prjemian So that is what workflow_helper is for? Connecting to PVs in the background?

prjemian commented 4 years ago

Right. Start the workflow_helper when spec is started.

On Sat, Sep 14, 2019, 3:00 PM qzhang234 notifications@github.com wrote:

So that is what workflow_helper is for? Connecting to PVs in the background?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/aps-8id-trr/ipython-8idiuser/issues/36?email_source=notifications&email_token=AARMUMH6MGDKSN7VZWL7DILQJU7GTA5CNFSM4IQEEG22YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6XC6OQ#issuecomment-531509050, or mute the thread https://github.com/notifications/unsubscribe-auth/AARMUMHYDQUFES4N66S7D2TQJU7GTANCNFSM4IQEEG2Q .

prjemian commented 4 years ago

Need one more number register to tell whether to use analysis (1) or transfer (0). 8idi:Reg172. Name it workflow_submit_xpcs_job

sureshnaps commented 4 years ago

ok that is a good plan. So did we decide that we do not need an extra json file?

prjemian commented 4 years ago

No JSON yet. First we assess how well this works.

sureshnaps commented 4 years ago

ok. it will be easier to first test it with spec using Lambda and calling the new .py code as that pipeline is already fully established.

prjemian commented 4 years ago

Also need the xpcs_qmap_file in a string register: xpcs_qmap_file in 8idi:StrReg13

sureshnaps commented 4 years ago

do you think it might be a good idea to put the qmap file with full path so we do not need to change the path or still prefer the hard coded /home/8-id-i/partitionMapLibrary/ and hackulated cycle dir and put only the qmap name?

If we use the hackulated cycle dir, then if we use this chain for post analysis at a later date, then it will fail.

prjemian commented 4 years ago

Your call. One way puts the risk at operations, the other puts the risk for post-analysis chain. I like the way you just suggested, put the qmap file with full path.

prjemian commented 4 years ago

BTW, I like the exercise to make this code work with both SPEC and bluesky. That process ensures we make a clean, common interface for both programs. The post-analysis chain will benefit, as long as this does not affect operations.

sureshnaps commented 4 years ago

ok that is good. We will use qmap with full path.

prjemian commented 4 years ago

Let's take good notes on that, exactly which 8idi:StrReg gets that value, so I can adjust the code. I should work that change in a branch form this branch and submit a PR back to this branch so we can see exactly what is the change.

prjemian commented 4 years ago

Here is the loop the helper runs: https://github.com/aps-8id-trr/ipython-8idiuser/blob/9a2c6258e697519ce8f8124374bd386f70c4ffcc/profile_bluesky/startup/spec_support/spec_DM_support.py#L197-L229

prjemian commented 4 years ago

current status of branch

[x] bluesky data structure written (35-DM-support.py : DataManagementMetadata)
[x] helper data structure written (spec_DM_support.py : DataManagementMetadata)
[x] helper routine written (spec_DM_support.py : WorkflowHelper)
[x] install logging code into helper
[ ] bluesky ad_acquire() updated with workflow_start, ... changes
[ ] use of qmap variable refactored

prjemian commented 4 years ago

I'm writing this TODO list since I must focus for the next few days on the APS-U/APS Advanced Controls Workshop on Tuesday. Back on Wednesday.

sureshnaps commented 4 years ago

ok to make it doable by us, can you add the syntax to call hdf from spec so we can test it on Monday.

prjemian commented 4 years ago

coming ... adding as docstring in the code

sureshnaps commented 4 years ago

thanks

prjemian commented 4 years ago

All that comes after the data acquisition, of course.

prjemian commented 4 years ago

@sureshnaps , @qzhang234 : I was able to start the workflow helper after a few code edits. You can test whether or not it works tomorrow.

Here is the comment from the code that tells you how to run it.

Run this command from the UNIX command line to
start the helper in the background:

    cd /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/spec_support
    helper &

It will emit log messages when a workflow is operating.
These messages will be saved in a file:
<pwd>/.logs/.workflow_helper.log
(pwd is the present working directory).
Once this code is found to be reliable, we can 
simplify the startup and running of this helper.

prjemian commented 4 years ago

Run the workflow helper daemon in the background:

tcsh:
- setenv HELPER_DIR /home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/spec_support
- setenv HELPER $HELPER_DIR/daemon.sh
bash:
- export HELPER_DIR=/home/beams/8IDIUSER/.ipython-bluesky/profile_bluesky/startup/spec_support
- export HELPER=$HELPER_DIR
start it: $HELPER start
stop it: $HELPER stop
restart it: $HELPER restart
Is it running? $HELPER status
log file: $HELPER_DIR/.logs/.workflow_helper.log

prjemian commented 4 years ago

Launch GUI tool to watch the register PVs.

First, pick the APSshare python3:

bash
source /APSshare/anaconda3/x86_64/bin/activate

Start the pvview tool:

pvview 8idi:StrReg1{0,1,2}.{DESC,VAL} 8idi:Reg17{0,1,2}.{DESC,VAL} &

aps-8id-dys / ipython-8idiuser