search

APS-USAXS / usaxs-bluesky-ended-2023

Bluesky instrument for USAXS
0 stars 0 forks source link

I somehow hanged the whole BlueSky #481

Closed jilavsky closed 3 years ago

jilavsky commented 3 years ago

I hanged this thing. Cannot even abort. When solving #480 I needed to stop. So I did ctrl-C. Eventually I got to prompt and type RE.abort(). That generated many errors "waiting for interlock" as I aborted in the middele of mode change. I used spec to move to defined state and issued RE.abort() again. I got in trouble again as nothing was happening. I did ctrl-C few times to get to Keyboard Interrupt. I tried exit and that failed also. Now it is hanging and no prompt. Seems like my only chance here is to kill the terminal tab?

''' ^CI Tue-21:23:08 - Waiting 5.67077 for PLC interlock, check limit switches I Tue-21:23:08 - Waiting 5.77195 for PLC interlock, check limit switches I Tue-21:23:08 - Waiting 5.87309 for PLC interlock, check limit switches I Tue-21:23:08 - Waiting 5.97428 for PLC interlock, check limit switches I Tue-21:23:08 - Waiting 6.07556 for PLC interlock, check limit switches I Tue-21:23:08 - Waiting 6.17706 for PLC interlock, check limit switches I Tue-21:23:08 - Waiting 6.27825 for PLC interlock, check limit switches I Tue-21:23:08 - Waiting 6.37946 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 6.4807 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 6.5819 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 6.68306 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 6.78431 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 6.88543 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 6.9867 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 7.08792 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 7.18917 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 7.2907 for PLC interlock, check limit switches I Tue-21:23:09 - Waiting 7.392 for PLC interlock, check limit switches I Tue-21:23:10 - Waiting 7.49312 for PLC interlock, check limit switches I Tue-21:23:10 - Waiting 7.59426 for PLC interlock, check limit switches I Tue-21:23:10 - Waiting 7.69553 for PLC interlock, check limit switches I Tue-21:23:10 - Waiting 7.79685 for PLC interlock, check limit switches ^CI Tue-21:23:10 - Waiting 7.89798 for PLC interlock, check limit switches I Tue-21:23:10 - Waiting 7.99906 for PLC interlock, check limit switches I Tue-21:23:10 - Waiting 8.10028 for PLC interlock, check limit switches ^CI Tue-21:23:10 - Waiting 8.20154 for PLC interlock, check limit switches I Tue-21:23:10 - Waiting 8.30312 for PLC interlock, check limit switches ^CI Tue-21:23:10 - Waiting 8.40428 for PLC interlock, check limit switches I Tue-21:23:11 - Waiting 8.5054 for PLC interlock, check limit switches I Tue-21:23:11 - Waiting 8.6066 for PLC interlock, check limit switches ^CI Tue-21:23:11 - Waiting 8.70779 for PLC interlock, check limit switches I Tue-21:23:11 - Waiting 8.80901 for PLC interlock, check limit switches ^CI Tue-21:23:11 - Waiting 8.91026 for PLC interlock, check limit switches I Tue-21:23:11 - Waiting 9.01162 for PLC interlock, check limit switches ^CI Tue-21:23:11 - Waiting 9.11294 for PLC interlock, check limit switches I Tue-21:23:11 - Waiting 9.21416 for PLC interlock, check limit switches ^CI Tue-21:23:11 - Waiting 9.31539 for PLC interlock, check limit switches I Tue-21:23:11 - Waiting 9.41663 for PLC interlock, check limit switches I Tue-21:23:12 - Waiting 9.51779 for PLC interlock, check limit switches ^CI Tue-21:23:12 - Waiting 9.61904 for PLC interlock, check limit switches I Tue-21:23:12 - Waiting 9.72032 for PLC interlock, check limit switches ^CI Tue-21:23:12 - Waiting 9.82274 for PLC interlock, check limit switches ^CI Tue-21:23:12 - Waiting 9.92422 for PLC interlock, check limit switches

KeyboardInterrupt Traceback (most recent call last) /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in handleWakeup(inp) 1527 os.close(wfd) 1528 -> 1529 def handleWakeup(inp): 1530 # here Python signal handler will be invoked 1531 # this book-keeping is to drain the pipe

/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in handler(signum, frame) 191 orig_func = self.original_handler 192 self.release() --> 193 orig_func(signum, frame) 194 195 self.handle_signals()

KeyboardInterrupt: Exception in thread bluesky-run-engine: Traceback (most recent call last): File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/threading.py", line 932, in _bootstrap_inner self.run() File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/threading.py", line 870, in run self._target(*self._args, **self._kwargs) File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/asyncio/base_events.py", line 570, in run_forever self._run_once() File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/asyncio/base_events.py", line 1823, in _run_once event_list = self._selector.select(timeout) File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/selectors.py", line 468, in select fd_event_list = self._selector.poll(timeout, max_ev) bluesky.run_engine._RunEnginePanic ^C--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func) 889 try: --> 890 self._during_task.block(self._blocking_event) 891 except KeyboardInterrupt:

/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in block(self, blocking_event) 1585 if vals[1] is not None: -> 1586 raise vals[1] 1587 finally:

/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in handleWakeup(inp) 1528 -> 1529 def handleWakeup(inp): 1530 # here Python signal handler will be invoked

/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/utils.py in handler(signum, frame) 192 self.release() --> 193 orig_func(signum, frame) 194

KeyboardInterrupt:

During handling of the above exception, another exception occurred:

KeyboardInterrupt Traceback (most recent call last)

in ----> 1 RE.abort() /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in abort(self, reason) 1112 :meth:`RunEngine.stop` 1113 """ -> 1114 return self.__interrupter_helper(self._abort_coro(reason)) 1115 1116 async def _abort_coro(self, reason): /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in __interrupter_helper(self, coro) 1196 coro_event.wait() 1197 if was_paused: -> 1198 self._resume_task() 1199 1200 return task.result() /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in _resume_task(self, init_func) 908 # (per decree of Guido) thus we give it a second 909 # to sort it's self out --> 910 task_finished = self._blocking_event.wait(1) 911 # before giving up and putting the RE in a 912 # non-recoverable panicked state. /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/threading.py in wait(self, timeout) 556 signaled = self._flag 557 if not signaled: --> 558 signaled = self._cond.wait(timeout) 559 return signaled 560 /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/threading.py in wait(self, timeout) 304 else: 305 if timeout > 0: --> 306 gotit = waiter.acquire(True, timeout) 307 else: 308 gotit = waiter.acquire(False) KeyboardInterrupt: In [11]: RE.abort() ^C--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) in ----> 1 RE.abort() /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in abort(self, reason) 1112 :meth:`RunEngine.stop` 1113 """ -> 1114 return self.__interrupter_helper(self._abort_coro(reason)) 1115 1116 async def _abort_coro(self, reason): /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/bluesky/run_engine.py in __interrupter_helper(self, coro) 1194 was_paused = self._state == 'paused' 1195 self.loop.call_soon_threadsafe(start_task) -> 1196 coro_event.wait() 1197 if was_paused: 1198 self._resume_task() /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/threading.py in wait(self, timeout) 556 signaled = self._flag 557 if not signaled: --> 558 signaled = self._cond.wait(timeout) 559 return signaled 560 /share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/threading.py in wait(self, timeout) 300 try: # restore state no matter what (e.g., KeyboardInterrupt) 301 if timeout is None: --> 302 waiter.acquire() 303 gotit = True 304 else: KeyboardInterrupt: In [12]: exit ''' Oh no, few ctrl-C and I killed BS: ''' ^CException ignored in: Traceback (most recent call last): File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/threading.py", line 1388, in _shutdown lock.acquire() KeyboardInterrupt: ^CException ignored in: ._remove at 0x7fb640c7f820> Traceback (most recent call last): File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/_weakrefset.py", line 38, in _remove ^CKeyboardInterrupt: ^CError in atexit._run_exitfuncs: Traceback (most recent call last): File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3783, in atexit_operations self.reset(new_session=False) File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 1431, in reset self.displayhook.flush() File "/share1/bluesky/conda_envs/bluesky_2021_1/lib/python3.8/site-packages/IPython/core/displayhook.py", line 311, in flush gc.collect() KeyboardInterrupt usaxs@usaxscontrol .../USAXS_data/2021-01 $ usaxs@usaxscontrol .../USAXS_data/2021-01 $ usaxs@usaxscontrol .../USAXS_data/2021-01 $ usaxs@usaxscontrol .../USAXS_data/2021-01 $ usaxs@usaxscontrol .../USAXS_data/2021-01 $ usaxs@usaxscontrol .../USAXS_data/2021-01 $ usaxs@usaxscontrol .../USAXS_data/2021-01 $ ''' This is lot of hurt to stop a scan...
prjemian commented 3 years ago

We replicated this same situation while working on #480. The only remedy we found was to force the ipython session to quit, pressing the [ctrl][\] keys together and waiting patiently. Less than a minute overall.