DM workflow failed for Lambda/Bluesky: File not transferred from /home/8ididata to /net/wolf

[qzhang234]

The DM analysis for Bluesky/Lambda failed at the file transfer stage. The files are saved on /home/8ididata but not on /net/wolf/. I confirmed that the workflow is correct by running the same thing in Spec and it works. Any suggestions what could be the problem?

Screen output from running Lambda with Bluesky

``` In [24]: In [24]: dm_workflow.transfer Out[24]: 'xpcs8-01-Lambda' In [25]: dm_workflow.analysis Out[25]: 'xpcs8-02-Lambda' In [26]: In [26]: RE(AD_Acquire(lambdadet, 'A003_Test', 0.1, 0.11, 100, '/home/8ididata/2020-3/demo202101')) Transient Scan ID: 203676 Time: 2021-01-11 16:35:14 Persistent Unique Scan ID: '485ceed8-5bd6-4d7a-b869-5f7fb52d14ad' New stream: 'baseline' New stream: 'Timebase_monitor' New stream: 'pind1_monitor' New stream: 'pind2_monitor' New stream: 'Atten1_monitor' New stream: 'Atten2_monitor' New stream: 'pid1_controlled_value_monitor' New stream: 'pid1_final_value_monitor' New stream: 'primary' +-----------+------------+ | seq_num | time | +-----------+------------+ | 1 | 16:35:26.7 | +-----------+------------+ generator AD_Acquire ['485ceed8'] (scan num: 203676) HDF5 workflow file name: /home/8ididata/2020-3/demo202101/A003_Test/A003_Test_0001-0100.hdf Out[26]: ('485ceed8-5bd6-4d7a-b869-5f7fb52d14ad',) In [27]: ```

[prjemian]

The only clues that Bluesky might have are in the ipython_logger files. What working directory are you using now? (This is not the data dir on /home/8ididata/2020-3)

[qzhang234]

It's /home/beams10/8IDIUSER/bluesky_data/2020/2020-3.

By the way I checked both the ipython_console.log and ipython_logger.log in ~/.logs, however the most recent record only goes up to 01/08. Not sure why.

[prjemian]

You might be looking at old content. Try closing the log file in your editor and then re-opening. Last lines in the console log are:

snow% tail ipython_console.log
#[Out]# 'xpcs8-01-Lambda'
# Mon, 11 Jan 2021 16:34:56
dm_workflow.analysis
#[Out]# 'xpcs8-02-Lambda'
# Mon, 11 Jan 2021 16:35:14
RE(AD_Acquire(lambdadet, 'A003_Test', 0.1, 0.11, 100, '/home/8ididata/2020-3/demo202101'))
#[Out]# ('485ceed8-5bd6-4d7a-b869-5f7fb52d14ad',)
# Mon, 11 Jan 2021 17:09:09
get_ipython().run_line_magic('pwd', '')
#[Out]# '/home/beams10/8IDIUSER/bluesky_data/2020/2020-3'
snow% wc -l ipython_console.log
95 ipython_console.log

[prjemian]

Also:

snow% pwd
/home/beams/8IDIUSER/bluesky_data/2020/2020-3/.logs
snow% grep 485cee *
ipython_console.log:#[Out]# ('485ceed8-5bd6-4d7a-b869-5f7fb52d14ad',)
snow% grep A003_Test *
ipython_console.log:RE(AD_Acquire(lambdadet, 'A003_Test', 0.1, 0.11, 100, '/home/8ididata/2020-3/demo202101'))
ipython_logger.log:|2021-01-11 16:35:14.772|INFO|17430|bluesky-session|xpcs_acquire|60|bluesky-run-engine| - file_path = /home/8ididata/2020-3/demo202101/A003_Test/
ipython_logger.log:|2021-01-11 16:35:15.642|INFO|17430|bluesky-session|lambda_750k|555|bluesky-run-engine| - full_name: /home/8ididata/2020-3/demo202101/A003_Test/A003_Test_00001-00100.imm
ipython_logger.log:|2021-01-11 16:35:27.086|INFO|17430|bluesky-session|xpcs_acquire|308|Thread-4147| - DM workflow kickoff starting: analysis:True  file:/home/8ididata/2020-3/demo202101/A003_Test/A003_Test_0001-0100.hdf

But none of this has any diagnostics about the DM workflow.

[prjemian]

What changed between the successful and unsuccessful runs shown above?

[qzhang234]

The fix you made earlier today allowed Bluesky to take data with Lambda, however none of the measurements went through the DM workflow. I thought it would be better to start another issue since the original one was already fixed and now the problem is somewhere else.

Sorry for the confusion.