search

APS-USAXS / usaxs-bluesky-ended-2023

Bluesky instrument for USAXS
0 stars 0 forks source link

After APS dump and restart, BS crashed out #579

Open jilavsky opened 1 year ago

jilavsky commented 1 year ago

AFter restarting from APS shut down we eventually (few minutes later) got Exception:

|       196 | 09:59:50.6 |    3000000 |      36164 |       2884 |                                                                                                      
|       197 | 09:59:51.2 |    3000000 |      40060 |       2943 |                                                                                                      
|       198 | 09:59:51.8 |    3000000 |      41680 |       2612 |                                                                                                      
|       199 | 09:59:52.6 |    6000000 |      90332 |       5103 |                                                                                                      
|       200 | 09:59:53.5 |    6000000 |      98754 |       5112 |                                                                                                      
|       201 | 09:59:54.4 |    6000000 |     109301 |       5291 |                                                                                                      
|       202 | 09:59:55.3 |    6000000 |     118587 |       5301 |                                                                                                      
|       203 | 09:59:56.1 |    6000000 |     127793 |       5420 |                                                                                                      
|       204 | 09:59:57.0 |    6000000 |     138288 |       5290 |                                                                                                      
|       205 | 09:59:57.9 |    6000000 |     146702 |       5584 |                                                                                                      
|       206 | 09:59:58.8 |    6000000 |     156572 |       5739 |                                                                                                      
+-----------+------------+------------+------------+------------+
generator uascan ['28ac729e'] (scan num: 914)
E Wed-10:00:00 - Exception FailedStatus during attempt 1 of 1 of command '('USAXSscan', ['20', '60', '1', 'GlassyCarbonM4'], 464, '      USAXSscan        20      60      1      "GlassyCarbonM4"')''
subject: FailedStatus during attempt 1 of 1 of command '('USAXSscan', ['20', '60', '1', 'GlassyCarbonM4'], 464, '      USAXSscan        20      60      1      "GlassyCarbonM4"')''

date: 2022-11-30 10:00:00.643779
command file: /share1/USAXS_data/2022-11/usaxs.mac
line number: 464
command: ('USAXSscan', ['20', '60', '1', 'GlassyCarbonM4'], 464, '      USAXSscan        20      60      1      "GlassyCarbonM4"')
raw command:       USAXSscan        20      60      1      "GlassyCarbonM4"
attempt: 1 of 1
exception: Status(obj=EpicsSignal(read_pv='9idcLAX:state', name='user_data_state', parent='user_data', value='counting 207/300', timestamp=1669823999.086156, auto_monitor=True, string=True, write_pv='9idcLAX:state', limits=False, put_complete=False), done=True, success=False)
Stopping further processing of this command list.

It is not clear if this is related to APS dump and why it would be, but it happened second time I remember AFTER APS dump and use of suspender. Makes me wonder, if suspender somwho does not impact this, eventually. Note, that this took may be 2-4 minutes before Excepting. Added here as note in case we keep seeing this that there maybe something wierd happening.

prjemian commented 1 year ago

and, again with updating the state PV: