Bluesky deployment 2022-04-20

JunAishima commented 2 years ago

see also slack discussion: https://nsls2.slack.com/archives/C01QJ5Y9UV8/p1650462398989789

JunAishima commented 2 years ago

ws1 - no .ipython directory under xf17id1 user. so we should do a Bluesky Full Deploy

Full Deploy worked well - /etc/profile.d/bluesky.sh looks correct
made a manual symlink from .ipython to profile_collection
Conda Base was not run separately as it is already part of Full Deploy
section "Update bluesky config and update to the new environments" done
nothing else in section "Profile collection maintenance" makes sense in a freshly pulled profile collection so need to move to the notebook directory now: (xf17id1-ca1) /epics/iocs/notebook/notebooks.
the profile_collection is located in /epics/iocs/notebook/.ipython/profile_collection
there are changes on branch 20220121-debugging - need to ask about the changes, the branch, and 17Q3_plus_new_ivu branch
- 20220121-debugging - lookup table changes, committed, this branch is to be merged into master and used for the next cycle. @AbbyGi and I to review the changes, @mrakitin to merge.
- 17Q3 branch - already incorporated, deleted the branch

AbbyGi commented 2 years ago

IOC ownership change completed successfully

JunAishima commented 2 years ago

profile_collection continued

tag 2022C1.1 installed and pushed as requested (after putting in the acceptance test - but done in the correct place)
acceptance test directory created, copy of notebook used for testing added into directory, commit has results to be compared with future expectations
awaiting approval and merging of 20220121-debugging branch, especially regarding notebook service permissions

JunAishima commented 2 years ago

continuing on

merged 20220121-debugging, pulled master on the already-existing shared profile_collection, and also onto the local notebook profile_collection - all profile_collections are synchronized, moved notebooks over to using the shared one
xf17id1-ws1:/home/xf17id1/.ipython/profile_collection - pointing to expected shared location
Update bluesky config appears to have been done - expected values in /etc/profile.d/bluesky.sh

kafka testing

multiple issues with bsui starting - ws1
- wrong backend - solved by commenting out widget backend type
- in 20-detectors.py - cameras are failing with a KeyError for "acquire" - implying that the acquire fields are not available for any of the cameras. very strange! comment 20-detectors.py and 95-custom_plans.py out for now
- still not able to run both the kafka monitoring script and a scan - getting
```
In [1]: import ophyd.sim
```

In [2]: hw = ophyd.sim.hw()

In [3]: RE(count([hw.det])) An exception has occurred, use '%tb verbose' to see the full traceback. ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 140532872914752 and this is thread id 140530181469952.

See /home/xf17id1/.cache/bluesky/log/bluesky.log for the full traceback.

JunAishima commented 2 years ago

Continued with Kafka testing

moved to using PersistentDict for metadata storage - see https://blueskyproject.io/bluesky/metadata.html#persistence-between-sessions for general directions. removed local restoring of metadata from SQLite file

tested Kafka publishing and it worked!


2022-04-21T17:21:54.804913 document: start
contents: {'beamline_id': 'AMX',
'detectors': ['det'],
'hints': {'dimensions': [[['time'], 'primary']]},
'num_intervals': 0,
'num_points': 1,
'plan_args': {'detectors': ["SynGauss(prefix='', name='det', "
                         "read_attrs=['val'], configuration_attrs=['Imax', "
                         "'center', 'sigma', 'noise', "
                         "'noise_multiplier'])"],
           'num': 1},
'plan_name': 'count',
'plan_type': 'generator',
'scan_id': 1,
'time': 1650576113.9515412,
'uid': '84c0cf17-1b14-4bb2-8ed2-b0c861830add',
'versions': {'bluesky': '1.8.3', 'ophyd': '1.6.4'}}

2022-04-21T17:21:54.805392 document: descriptor contents: {'configuration': {'det': {'data': {'det_Imax': 1, 'det_center': 0, 'det_noise': 'none', 'det_noise_multiplier': 1, 'det_sigma': 1}, 'data_keys': {'det_Imax': {'dtype': 'integer', 'shape': [], 'source': 'SIM:det_Imax'}, 'det_center': {'dtype': 'integer', 'shape': [], 'source': 'SIM:det_center'}, 'det_noise': {'dtype': 'integer', 'enum_strs': ['none', 'poisson', 'uniform'], 'shape': [], 'source': 'SIM:det_noise'}, 'det_noise_multiplier': {'dtype': 'integer', 'shape': [], 'source': 'SIM:det_noise_multiplier'}, 'det_sigma': {'dtype': 'integer', 'shape': [], 'source': 'SIM:det_sigma'}}, 'timestamps': {'det_Imax': 1650576110.373778, 'det_center': 1650576110.37377, 'det_noise': 1650576110.3737009, 'det_noise_multiplier': 1650576110.373756, 'det_sigma': 1650576110.3737867}}}, 'data_keys': {'det': {'dtype': 'number', 'object_name': 'det', 'precision': 3, 'shape': [], 'source': 'SIM:det'}}, 'hints': {'det': {'fields': ['det']}}, 'name': 'primary', 'object_keys': {'det': ['det']}, 'run_start': '84c0cf17-1b14-4bb2-8ed2-b0c861830add', 'time': 1650576114.1404076, 'uid': 'd442fd89-4b7f-4f9f-83be-b62c757ae3d9'}

2022-04-21T17:21:54.806199 document: event contents: {'data': {'det': 1.0}, 'descriptor': 'd442fd89-4b7f-4f9f-83be-b62c757ae3d9', 'filled': {}, 'seq_num': 1, 'time': 1650576114.2503617, 'timestamps': {'det': 1650576114.1391304}, 'uid': 'f0f5864b-95ce-44eb-b702-ca4dc31b3321'}

2022-04-21T17:21:54.806328 document: stop contents: {'exit_status': 'success', 'num_events': {'primary': 1}, 'reason': '', 'run_start': '84c0cf17-1b14-4bb2-8ed2-b0c861830add', 'time': 1650576114.2909095, 'uid': 'cef84d0d-6f61-4676-b773-7bfa11765beb'}

JunAishima commented 2 years ago

fix-ci pull request merged to help with timeout issues
test using xf17id1 user for notebook to make acceptance test run on notebooks
- modified /etc/systemd/system/jupyter-amx.service to use xf17id1:xf17id1 - also needed to change ownership of /var/log/jupyter to xf17id1:xf17id1
- dzdo systemctl restart jupyter-amx
- (change into /epics/iocs/notebook)
- dzdo chown xf17id1:xf17id1 * -R
- dzdo chown xf17id1:xf17id1 .config -R
- dzdo chown xf17id1:xf17id1 .local -R
- dzdo chown xf17id1:xf17id1 .ipython -R
run jupyter notebook for acceptance test 2022_04_20_BlueskyDeployment.ipynb
- Notebook doesn't seem to run startup files - not sure why (nothing in logs)
manually running commands in notebook in bsui
- wa fails - debugging (thanks @AbbyGi ) shows the following source of the problem:
- ipdb> p p FESlitsCenter(prefix='FE:C17A-OP{Slt:12-Ax:X}', name='fe_x_ctr', parent='fe', settle_time=0.0, timeout=None, read_attrs=['setpoint', 'readback', 'stop_signal'], configuration_attrs=[], limits=None, egu='')
- I cannot get any response from the PVs expected here, such as FE:C17A-OP{Slt:12-Ax:X}.center
- Stu Myers mentions that there is talk about accelerator IOCs being worked on - if the IOC controlling these slits is down, we cannot do acceptance tests now.
RE(vdcm_rock_test()) scan runs in bsui! however, the analysis doesn't seem to be working (but with no signal, no surprise)

JunAishima commented 2 years ago

remaining:

wa in bsui or notebook - needs to be done once accelerator IOCs are back, to be able to get valid values from front end slits
merging 2022-c2 (check branch name) code into master
make a note that the acceptance test has been modified to use bsui, since we can't get the notebook to run successfully
notebook work including why startup files aren't being run (Tuesday)

JunAishima commented 2 years ago

wa now works!

In [1]: wa
/nsls2/conda/envs/2022-2.1-py39-tiled/lib/python3.9/site-packages/bluesky/magics.py:34: UserWarning: BlueskyMagics.positioners is deprecated. Please use the newer labels feature.
  warnings.warn("BlueskyMagics.positioners is deprecated. "
Positioner                     Value       Low Limit   High Limit  Offset     
atten_a1                       -12.3       -999999999.3 10000000.7  -12.3      
atten_a2                       -12.1       -12.1       -12.1       -12.1      
atten_a3                       -12.5       -12.5       -12.5       -12.5      
atten_a4                       -12.8       -12.8       -12.8       -12.8      
bs_fx                          -1179.997   0.0         0.0         0.0        
bs_fy                          -870.0      -9999999999.0 999999999.0 0.0        
colli_x                        1820.0      400.0       400.0       400.0      
colli_xy                       16672.4     0.0         0.0         0.0        
colli_z                        -225.0      0.0         0.0         0.0        
fe_b                           -1.69996    -10.0       10.0        0.0        
fe_i                           -0.34984    -10.0       10.0        0.0        
fe_o                           2.14984     -10.0       10.0        0.0        
fe_t                           1.80004     -9.9        10.1        0.1        
fe_x_ctr                       0.9         -10.0       10.0        AttributeError
fe_x_gap                       2.49968     -20.0       20.0        AttributeError
fe_y_ctr                       0.05004     -9.95       10.05       AttributeError
fe_y_gap                       3.5         -19.9       20.1        AttributeError
gonio_gx                       231.83      -16000.0    16000.0     0.0        
gonio_gy                       -0.46       0.0         0.0         0.0        
gonio_gz                       -5.47       0.0         0.0         0.0        
gonio_o                        61.446528   -35865.0    36135.0     60.0       
gonio_py                       -922.117    -5000.0     5000.0      -922.217   
gonio_pz                       0.2         -5000.0     5000.0      0.0        
hhls_b                         -2.4        -0.31       -0.31       -0.31      
hhls_i                         -1.599      -2.36       -2.36       -2.36      
hhls_o                         2.1         -3.6        -3.6        -3.6       
hhls_t                         1.8         -0.19       -0.19       -0.19      
hhls_x_ctr                     -2.98       -2.98       -2.98       -2.98      
hhls_x_gap                     -1.24       -1.24       -1.24       -1.24      
hhls_y_ctr                     -0.25       -0.25       -0.25       -0.25      
hhls_y_gap                     0.12        0.12        0.12        0.12       
hm_x                           1050.0      0.0         0.0         0.0        
hm_y                           -150.0      0.0         0.0         0.0        
hm_z                           -57.83      0.0         0.0         0.0        
ivu_gap_gap                    25150.0     6200.0      25200.0     0.0        
kbm_hp                         3.458       4.088       4.088       4.088      
kbm_hr                         -2.5        -12.195     -12.195     -12.195    
kbm_hx                         -1.846      -7.0        9.0         7.456      
kbm_hy                         0.154       -0.828      -0.828      -0.828     
kbm_vp                         -3.672      -5.0        2.0         1.53       
kbm_vx                         0.0         5.75        5.75        5.75       
kbm_vy                         1.845       6.3         6.3         6.3        
mbpm1_x                        739.28      -5400.0     5400.0      0.0        
mbpm1_y                        -1067.06    -9700.0     5100.0      0.0        
mbpm2_x                        0.01        0.0         0.0         0.0        
mbpm2_y                        -3.78       0.0         0.0         0.0        
mbpm3_x                        -237.0      -237.0      -237.0      -237.0     
mbpm3_y                        -526.87     -526.88     -526.88     -526.88    
mic_x                          285.98      0.0         0.0         0.0        
mic_y                          -300.08     -100000.0   100000.0    0.0        
sht_x                          1100.0      -5900.0     14500.0     1100.0     
slits1_b                       -1299.99    0.0         0.0         0.0        
slits1_i                       -1500.01    0.0         0.0         0.0        
slits1_o                       1500.04     0.0         0.0         0.0        
slits1_t                       1500.02     0.0         0.0         0.0        
slits1_x_ctr                   0.0         0.0         0.0         0.0        
slits1_x_gap                   2999.97     0.0         0.0         0.0        
slits1_y_ctr                   100.01      0.0         0.0         0.0        
slits1_y_gap                   2800.01     0.0         0.0         0.0        
slits2_b                       -3725.01    -3653.83    -3653.83    -3653.83   
slits2_i                       -825.01     -861.46     -861.46     -861.46    
slits2_o                       4549.99     4361.46     4361.46     4361.46    
slits2_t                       1200.01     1220.17     1220.17     1220.17    
slits2_x_ctr                   1862.5      1750.01     1750.01     1750.01    
slits2_x_gap                   5375.0      5222.92     5222.92     5222.92    
slits2_y_ctr                   -1262.5     -1216.83    -1216.83    -1216.83   
slits2_y_gap                   4925.01     4873.99     4873.99     4873.99    
slits3_b                       -2180.0     -2180.0     -2180.0     -2180.0    
slits3_i                       -205.0      -205.0      -205.0      -205.0     
slits3_o                       200.0       200.0       200.0       200.0      
slits3_t                       -2410.0     -2410.0     -2410.0     -2410.0    
slits3_x_ctr                   -2.5        -2.5        -2.5        -2.5       
slits3_x_gap                   405.0       405.0       405.0       405.0      
slits3_y_ctr                   -2295.0     -2295.0     -2295.0     -2295.0    
slits3_y_gap                   -230.0      -230.0      -230.0      -230.0     
slits4_b                       0.0         0.0         0.0         0.0        
slits4_i                       0.0         0.0         0.0         0.0        
slits4_o                       0.0         0.0         0.0         0.0        
slits4_t                       0.0         0.0         0.0         0.0        
slits4_x_ctr                   0.0         0.0         0.0         0.0        
slits4_x_gap                   0.0         0.0         0.0         0.0        
slits4_y_ctr                   0.0         0.0         0.0         0.0        
slits4_y_gap                   0.0         0.0         0.0         0.0        
tdm_pd                         3.167776    2.262       2.262       2.262      
tdm_pu                         3.527932    -100000.0   100000.0    4.04       
tdm_rd                         6.5         -10.747     -10.747     -10.747    
tdm_xd                         1.023       1.04        1.04        1.04       
tdm_xu                         1.152       1.32        1.32        1.32       
tdm_yd                         -6.001      -100.0      100.0       1.603      
tdm_yu                         -6.0        4.1         4.1         4.1        
vdcm_b                         8.437       -8.423      23.504      -0.381     
vdcm_e                         13475.4     0.0         0.0         0.0        
vdcm_g                         14.739      -999.0      999.0       17.66      
vdcm_p                         6.626669    -5.619      -5.619      -5.619     
vdcm_r                         5.774       -211.969    188.031     -19.822    
vdcm_w                         DisconnectedError

JunAishima commented 2 years ago

Notebook work:

clean up files in /epics/iocs
make /home/xf17id1 - copy skel files into ., .ipython directory

future notes to staff:

notebooks to be put into git, then pushed onto nsls2 or another safe site for future reference
matplotlib widget - see whether explicitly setting is necessary. would like to remove
lacking 2022-2.0-py39 conda env - need to run configuration
populated services with jupyter-amx and st.cmd
extract kernels - link from /usr/share/jupyter -> /epics/iocs/notebook/notebooks/kernels
nbconvert - classic notebooks (not necessary?)
kernels 2022-2.2-py39-tiled, 2022-2.1-py39-tiled, 2022-2.0-py39 - all will be available
kernels-available directory made - allow for some old ones to be not visible in Jupyter, but still available by a simple symlink

after cleanup, processes owned by xf17id1

log file directory owned by xf17id1 - as we want

now add password

create config - jupyter server config?
now in /home/xf17id1/.jupyter/jupyter_server_config.py
stored in Bitwarden under n2sn-inststaff-amx

JunAishima commented 2 years ago

start notebook - error during startup but hard to find in logging
use 2022-2.0-py39 kernel - starts up, and wa works!
- however, we do not get plotting working on vdcm_rock() - Javascript errors with Ipython - guess this is the lab interface?
classic notebook interface - plotting worked? I am very very confused

JunAishima commented 2 years ago

updating to 2022-2.2-py39-tiled - ws1 - done, checked BS_ENV, looks good
checked iocs ioc1, ioc2, srv1 - all seem ok with running services owned by softioc-amx
both ws1 xf17id1:/.ipython/profile_collection and xf17id1-ca1:/epics/iocs/notebook/notebooks/startup point to the shared copy. fixed link of -ca1 so that it goes directly to /nsls2/data/amx/shared/config/bluesky/profile_collection/startup instead of via ../.ipython/profile_collection * notebook acceptance test - with 2.2 kernel, startup 00-base fails - I really should wait until Maksim is happy and we can declare that we are going to all 2.3 - DO NOT WORRY ABOUT THIS FOR NOW
notebook acceptance test - go back to 2.0 kernel - it works
kafka test - runs as expected with 2.2
alarm mode normal

JunAishima commented 2 years ago

plan:

add 2.3 kernel to notebook
re-do acceptance test with 2.3
if passing, make 2.3 the default

Edit: scrap all of these - other beamlines have standardized to 2.2 and we are already there.

Successfully merged profile collection changes with passing CI. Done!

NSLS-II-AMX / profile_collection

Bluesky deployment 2022-04-20 #11