search

NSLS-II / Bug-Reports

Unified issue-tracker for bugs in the data acquisition, management, and analysis software at NSLS-II
BSD 3-Clause "New" or "Revised" License
2 stars 5 forks source link

Death of BSUI #177

Closed cmazzoli closed 6 years ago

cmazzoli commented 6 years ago

I report it for future memory as it died in a quite of a funny way, I believe (see below)..

In [331]: RE(dscan(sx,-.1,0.1,2))
Transient Scan ID: 82516 @ 2017/08/03 06:45:21
Persistent Unique Scan ID: 'ce9ba555-13f0-4102-8a3b-95cf0f57c6eb'
+-----------+------------+------------+------------------+-------------------+-------------------+-------------------+------------+------------+------------+
|   seq_num |       time |         sx | sx_user_setpoint | fccd_stats2_total | fccd_stats3_total | fccd_stats4_total |   sclr_ch3 |     temp_a |     temp_b |
+-----------+------------+------------+------------------+-------------------+-------------------+-------------------+------------+------------+------------+
|         1 | 06:47:49.9 |   4.999999 |         4.999999 |           2149597 |           1473102 |           5049284 |          0 |     19.990 |     23.019 |
|         2 | 06:48:09.0 |   5.099999 |         5.099999 |           2029589 |           1459694 |           5466325 |          0 |     20.011 |     23.025 |
|         3 | 06:48:30.4 |   5.199999 |         5.199999 |           2755150 |           1748933 |           7779179 |          0 |     20.012 |     23.025 |
+-----------+------------+------------+------------------+-------------------+-------------------+-------------------+------------+------------+------------+
generator dscan ['ce9ba5'] (scan num: 82516)
Out[331]: ['ce9ba555-13f0-4102-8a3b-95cf0f57c6eb']

In [332]: plt.figure()
Out[332]: <matplotlib.figure.Figure at 0x7f5d2a17f898>

In [333]: RE(d2scan(theta,-1*rangelll,1*rangelll,delta,-2.4793*1*rangelll,2.4793*1.0*rangelll,60))
[1]+  Stopped                 bsui/03 06:53:20
[xf23id1@xf23id-ws3 ~/Users/2017_08_Li]$ cd 2017_07
bash: cd: 2017_07: No such file or directory----------------+------------+---------------------+-------------------+-------------------+-------------------+------------+------------+------------+
[xf23id1@xf23id-ws3 ~/Users/2017_08_Li]$ heta_user_setpoint |      delta | delta_user_setpoint | fccd_stats2_total | fccd_stats3_total | fccd_stats4_total |   sclr_ch3 |     temp_a |     temp_b |
[xf23id1@xf23id-ws3 ~/Users/2017_08_Li]$ -------------------+------------+---------------------+-------------------+-------------------+-------------------+------------+------------+------------+
[xf23id1@xf23id-ws3 ~/Users/2017_08_Li]$            48.2983 |    75.5303 |             75.5303 |            189290 |             48170 |           3884549 |          0 |     29.984 |     32.033 |
[xf23id1@xf23id-ws3 ~/Users/2017_08_Li]$            48.4049 |    75.7947 |             75.7947 |            150695 |             38175 |           3017652 |          0 |     29.998 |     32.047 |
[xf23id1@xf23id-ws3 ~/Users/2017_08_Li]$            48.5116 |    76.0592 |             76.0592 |             78786 |             27042 |           2022214 |          0 |     29.992 |     32.054 |
[xf23id1@xf23id-ws3 ~/Users/2017_08_Li]$            48.6182 |    76.3236 |             76.3237 |            162128 |             42389 |           3771927 |          0 |     30.002 |     32.060 |
[xf23id1@xf23id-ws3 ~/Users/2017_08_Li]$            48.7249 |    76.5881 |             76.5881 |            192917 |             53105 |           4177258 |          0 |     30.001 |     32.068 |
|         6 | 06:54:35.4 |    48.8316 |             48.8316 |    76.8526 |             76.8526 |            151878 |             40623 |           3422236 |          0 |     29.996 |     32.070 |
|         7 | 06:54:45.5 |    48.9383 |             48.9383 |    77.1170 |             77.1170 |            191010 |             45209 |           3679725 |          0 |     30.003 |     32.073 |
|         8 | 06:54:55.6 |    49.0449 |             49.0449 |    77.3815 |             77.3815 |            121236 |             30919 |           2651498 |          0 |     30.000 |     32.077 |
|         9 | 06:55:05.9 |    49.1516 |             49.1516 |    77.6459 |             77.6459 |            174246 |             44850 |           3628729 |          0 |     30.000 |     32.083 |
|        10 | 06:55:16.3 |    49.2582 |             49.2583 |    77.9104 |             77.9104 |            185789 |             45530 |           3774577 |          0 |     30.012 |     32.081 |
|        11 | 06:55:26.5 |    49.3649 |             49.3649 |    78.1749 |             78.1749 |             94337 |             23526 |           1964252 |          0 |     29.998 |     32.081 |
|        12 | 06:55:37.0 |    49.4716 |             49.4716 |    78.4393 |             78.4393 |            145062 |             36898 |           3298366 |          0 |     30.004 |     32.083 |
|        13 | 06:55:47.3 |    49.5783 |             49.5783 |    78.7038 |             78.7038 |            117017 |             31799 |           2494725 |          0 |     30.007 |     32.088 |
|        14 | 06:55:57.4 |    49.6849 |             49.6849 |    78.9683 |             78.9682 |            148785 |             38527 |           2786336 |          0 |     30.007 |     32.086 |
|        15 | 06:56:07.7 |    49.7916 |             49.7916 |    79.2327 |             79.2327 |            179971 |             47511 |           3689200 |          0 |     29.998 |     32.088 |
|        16 | 06:56:17.8 |    49.8983 |             49.8983 |    79.4972 |             79.4972 |            180810 |             49810 |           3789123 |          0 |     30.003 |     32.089 |
|        17 | 06:56:28.5 |    50.0049 |             50.0049 |    79.7616 |             79.7616 |            151794 |             41864 |           3062729 |          0 |     29.997 |     32.086 |
|        18 | 06:56:44.3 |    50.1116 |             50.1116 |    80.0261 |             80.0261 |            249456 |             67964 |           4707320 |          0 |     30.010 |     32.093 |
|        19 | 06:56:57.1 |    50.2183 |             50.2183 |    80.2905 |             80.2905 |            165917 |             50383 |           3021611 |          0 |     30.004 |     32.090 |
|        20 | 06:57:07.5 |    50.3249 |             50.3249 |    80.5550 |             80.5550 |            177921 |             60206 |           2892944 |          0 |     30.011 |     32.091 |
|        21 | 06:57:17.8 |    50.4316 |             50.4316 |    80.8194 |             80.8194 |            195969 |             68312 |           2768339 |          0 |     29.998 |     32.092 |
|        22 | 06:57:28.7 |    50.5383 |             50.5383 |    81.0839 |             81.0839 |            232757 |             93538 |           3175899 |          0 |     30.001 |     32.087 |
|        23 | 06:57:38.9 |    50.6449 |             50.6449 |    81.3484 |             81.3484 |            229318 |            104305 |           2813070 |          0 |     30.008 |     32.092 |
|        24 | 06:57:49.2 |    50.7516 |             50.7516 |    81.6128 |             81.6128 |            316286 |            145302 |           3529014 |          0 |     30.001 |     32.093 |
|        25 | 06:57:59.6 |    50.8583 |             50.8582 |    81.8773 |             81.8773 |            421005 |            206569 |           3548900 |          0 |     30.002 |     32.092 |
|        26 | 06:58:10.4 |    50.9649 |             50.9649 |    82.1417 |             82.1417 |            454208 |            270162 |           2306051 |          0 |     29.989 |     32.088 |
|        27 | 06:58:21.1 |    51.0716 |             51.0716 |    82.4062 |             82.4062 |            728834 |            435868 |           3962465 |          0 |     30.006 |     32.090 |
|        28 | 06:58:31.3 |    51.1783 |             51.1783 |    82.6707 |             82.6707 |            982986 |            651533 |           3553070 |          0 |     30.003 |     32.089 |
|        29 | 06:58:41.8 |    51.2849 |             51.2849 |    82.9351 |             82.9351 |           1400014 |            974644 |           4262261 |          0 |     30.000 |     32.088 |
|        30 | 06:58:52.5 |    51.3916 |             51.3916 |    83.1996 |             83.1996 |           1764991 |           1266476 |           4364834 |          0 |     30.011 |     32.094 |
|        31 | 06:59:03.4 |    51.4983 |             51.4983 |    83.4640 |             83.4640 |           1713747 |           1259477 |           3982671 |          0 |     30.005 |     32.091 |
|        32 | 06:59:13.7 |    51.6049 |             51.6049 |    83.7285 |             83.7285 |           1374353 |            944539 |           3818473 |          0 |     30.002 |     32.094 |
|        33 | 06:59:24.6 |    51.7116 |             51.7116 |    83.9930 |             83.9930 |            966081 |            593223 |           4796981 |          0 |     30.001 |     32.089 |
|        34 | 06:59:35.2 |    51.8183 |             51.8183 |    84.2574 |             84.2574 |            598062 |            363305 |           2792401 |          0 |     30.001 |     32.091 |
|        35 | 06:59:48.7 |    51.9249 |             51.9249 |    84.5219 |             84.5219 |            436960 |            250119 |           2941897 |          0 |     29.994 |     32.089 |
|        36 | 07:00:01.6 |    52.0316 |             52.0316 |    84.7863 |             84.7863 |            365010 |            184798 |           3514480 |          0 |     30.000 |     32.093 |
|        37 | 07:00:12.5 |    52.1383 |             52.1382 |    85.0508 |             85.0508 |            352216 |            156441 |           4356085 |          0 |     29.996 |     32.092 |
|        38 | 07:00:22.8 |    52.2449 |             52.2449 |    85.3152 |             85.3152 |            271432 |            116241 |           3558448 |          0 |     30.001 |     32.090 |
|        39 | 07:00:33.2 |    52.3516 |             52.3516 |    85.5797 |             85.5797 |            283148 |            108365 |           4298997 |          0 |     30.004 |     32.093 |
|        40 | 07:00:43.5 |    52.4583 |             52.4583 |    85.8442 |             85.8442 |            131638 |             62890 |           1860400 |          0 |     30.004 |     32.093 |
|        41 | 07:00:53.7 |    52.5649 |             52.5649 |    86.1086 |             86.1086 |            165231 |             67933 |           2381064 |          0 |     30.000 |     32.093 |
|        42 | 07:01:04.0 |    52.6716 |             52.6716 |    86.3731 |             86.3731 |            216694 |             78371 |           3461155 |          0 |     30.001 |     32.095 |
|        43 | 07:01:14.6 |    52.7783 |             52.7783 |    86.6375 |             86.6375 |            163235 |             60321 |           3173619 |          0 |     30.002 |     32.095 |
|        44 | 07:01:24.8 |    52.8849 |             52.8849 |    86.9020 |             86.9020 |            196481 |             66555 |           3389299 |          0 |     29.995 |     32.090 |
|        45 | 07:01:35.7 |    52.9916 |             52.9916 |    87.1665 |             87.1665 |            169180 |             55323 |           2996052 |          0 |     30.006 |     32.092 |
|        46 | 07:01:46.1 |    53.0983 |             53.0983 |    87.4309 |             87.4309 |            219698 |             67729 |           3918778 |          0 |     29.999 |     32.094 |
|        47 | 07:01:56.2 |    53.2049 |             53.2049 |    87.6954 |             87.6954 |            142519 |             46476 |           2528695 |          0 |     30.008 |     32.092 |
|        48 | 07:02:06.4 |    53.3116 |             53.3116 |    87.9598 |             87.9598 |            139695 |             46840 |           2978211 |          0 |     30.002 |     32.095 |
|        49 | 07:02:16.6 |    53.4183 |             53.4183 |    88.2243 |             88.2243 |            154376 |             46309 |           3174288 |          0 |     29.998 |     32.097 |
+-----------+------------+------------+---------------------+------------+---------------------+-------------------+-------------------+-------------------+------------+------------+------------+
|   seq_num |       time |      theta | theta_user_setpoint |      delta | delta_user_setpoint | fccd_stats2_total | fccd_stats3_total | fccd_stats4_total |   sclr_ch3 |     temp_a |     temp_b |
+-----------+------------+------------+---------------------+------------+---------------------+-------------------+-------------------+-------------------+------------+------------+------------+
|        50 | 07:02:26.6 |    53.5249 |             53.5249 |    88.4887 |             88.4887 |            109274 |             37155 |           2215144 |          0 |     29.996 |     32.093 |
|        51 | 07:02:36.9 |    53.6316 |             53.6316 |    88.7532 |             88.7532 |            103677 |             30678 |           2245980 |          0 |     30.009 |     32.094 |
|        52 | 07:02:47.4 |    53.7383 |             53.7383 |    89.0177 |             89.0177 |            223255 |             59368 |           4355751 |          0 |     30.002 |     32.094 |
|        53 | 07:03:07.8 |    53.8449 |             53.8449 |    89.2821 |             89.2821 |             82134 |             27279 |           1632782 |          0 |     29.998 |     32.093 |
|        54 | 07:03:18.7 |    53.9516 |             53.9516 |    89.5466 |             89.5466 |            105642 |             33751 |           2359489 |          0 |     30.001 |     32.097 |
|        55 | 07:03:28.9 |    54.0583 |             54.0582 |    89.8110 |             89.8110 |            125400 |             35697 |           2670260 |          0 |     29.994 |     32.092 |
|        56 | 07:03:39.1 |    54.1649 |             54.1649 |    90.0755 |             90.0755 |            136110 |             37742 |           2781708 |          0 |     30.003 |     32.094 |
|        57 | 07:03:49.6 |    54.2716 |             54.2716 |    90.3400 |             90.3400 |            187997 |             51452 |           3927844 |          0 |     30.001 |     32.091 |
|        58 | 07:03:59.9 |    54.3783 |             54.3783 |    90.6044 |             90.6044 |            157024 |             40956 |           3115108 |          0 |     30.009 |     32.093 |
|        59 | 07:04:10.0 |    54.4849 |             54.4849 |    90.8689 |             90.8689 |            122021 |             36370 |           2513547 |          0 |     29.994 |     32.093 |
|        60 | 07:04:20.9 |    54.5916 |             54.5916 |    91.1333 |             91.1333 |            213202 |             53831 |           4121621 |          0 |     29.992 |     32.093 |
|        61 | 07:04:31.3 |    54.6983 |             54.6983 |    91.3978 |             91.3978 |            107864 |             31051 |           2365177 |          0 |     30.006 |     32.094 |
+-----------+------------+------------+---------------------+------------+---------------------+-------------------+-------------------+-------------------+------------+------------+------------+
generator d2scan ['53c458'] (scan num: 82517)
Out[333]: ['53c458da-2077-492d-8ce1-2b916cfa2c0a']

[xf23id1@xf23id-ws3 ~/Users/2017_08_Li]$ bsui
discarding /opt/conda/bin from PATH
prepending /opt/conda_envs/collection-17Q2.0/bin to PATH
Activating auto-logging. Current session state plus future input saved.
Filename       : /epics/xf/23id/ophyd/logs/xf23id1_log.py
Mode           : append
Output logging : False
Raw input log  : False
Timestamping   : False
State          : active
Python 3.6.1 |Continuum Analytics, Inc.| (default, Mar 22 2017, 19:54:23) 
Type "copyright", "credits" or "license" for more information.

IPython 5.3.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

IPython profile: collection
Using matplotlib backend: Qt5Agg
Loading metadata history from /home/xf23id1/.config/bluesky/bluesky_history.db

Enabling ring current suspender with 200 mA threshold and 120 s recovery time

Enabling FE shutter suspender with 15*60 s recovery time

Preparing CSX1 Photon Shutter 1 suspender with 3*60 s recovery time
    RE.install_suspender(ps1_shut_suspender) to enable.

In [1]: olog('Scan #{} Lscan @ {:.3f} eV, {:.2f} K, {:.2f} Hz'.format((db[-1].get('start').get('scan_id')+1),pgm_en.readback.value, temp.a.value,1,))
tacaswell commented 6 years ago

did you hit ctrl-z?

cmazzoli commented 6 years ago

My users reported that it died by itself.. Claudio

On Aug 3, 2017 9:40 PM, "Thomas A Caswell" notifications@github.com wrote:

did you hit ctrl-z?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/NSLS-II/Bug-Reports/issues/177#issuecomment-320133702, or mute the thread https://github.com/notifications/unsubscribe-auth/AKjlFjxPPNzcBTgMu7YgAZYFWTZ_Ut06ks5sUnZ-gaJpZM4OspCa .

tacaswell commented 6 years ago

The really weird thing is that it looks like python is suspended / backgrounded, they did something in bash, and then the table started to print and then they restarted so you have have a zombie bluesky floating around....

cmazzoli commented 6 years ago

Mmmm.. I may try to investigate better but the knowledge level is low enough, I believe, that they probably don't know/noticed what they actually did.. Anyway, any advice on how to solve/mitigate the problem? Should I ps aux | grep ipython or similar and try to kill the old job if any?

tacaswell commented 6 years ago

Yes, that is probably a good idea.

We talked about also capturing the signal that Ctrl-z sends, but decided against it to leave us with an escape hatch.

danielballan commented 6 years ago

It looks like this was resolved -- or as resolves as it can be. Please re-open if I'm wrong about that.