search

helgestein / helao-pub

hierachical automation of the natural sciences
MIT License
20 stars 5 forks source link

connection closed abnormally #13

Closed Yohko closed 3 years ago

Yohko commented 3 years ago

When only using async-orch and any server (server alone starts ok):

(helao) E:\Matthias\GitHub\helao-dev>python helao.py world 'pids_world.pck' does not exist, writing empty global dict. Configuration for 'world' is valid. Launching server group. Launching potentiostat at 127.0.0.1:8003 using server/gamry_server.py Gamry simulator loaded. ←[32mINFO←[0m: Started server process [←[36m1964←[0m] ←[32mINFO←[0m: Waiting for application startup. ←[32mINFO←[0m: Application startup complete. ←[32mINFO←[0m: Uvicorn running on ←[1mhttp://127.0.0.1:8003←[0m (Press CTRL+C to quit) Launching action group. Launching orchestrators group. Launching orchestrator at 127.0.0.1:8010 using orchestrators/async_orch.py ←[32mINFO←[0m: Started server process [←[36m5288←[0m] ←[32mINFO←[0m: Waiting for application startup. ←[32mINFO←[0m: 127.0.0.1:49311 - "←[1mPOST /potentiostat/get_status HTTP/1.1←[0m" ←[32m200 OK←[0m Orch decision length: 3 ←[32mINFO←[0m: Application startup complete. ←[32mINFO←[0m: Uvicorn running on ←[1mhttp://127.0.0.1:8010←[0m (Press CTRL+C to quit) ←[32mINFO←[0m: ('127.0.0.1', 49312) - "WebSocket /potentiostat/ws_status" [accepted] Launching visualizer group. CTRL-x to terminate process group. CTRL-d to disconnect. [('potentiostat', '127.0.0.1', 8003, 1964), ('orchestrator', '127.0.0.1', 8010, 5288)] Task exception was never retrieved future: <Task finished name='Task-4' coro=<OrchHandler.handle_socket() done, defined at E:\Matthias\GitHub\helao-dev\core\classes.py:90> exception=ConnectionClosedError('code = 1006 (connection closed abnormally [internal]), no reason')> Traceback (most recent call last): File "C:\ProgramData\Miniconda3\envs\helao\lib\site-packages\websockets\protocol.py", line 827, in transfer_data message = await self.read_message() File "C:\ProgramData\Miniconda3\envs\helao\lib\site-packages\websockets\protocol.py", line 895, in read_message frame = await self.read_data_frame(max_size=self.max_size) File "C:\ProgramData\Miniconda3\envs\helao\lib\site-packages\websockets\protocol.py", line 971, in read_data_frame frame = await self.read_frame(max_size) File "C:\ProgramData\Miniconda3\envs\helao\lib\site-packages\websockets\protocol.py", line 1047, in read_frame frame = await Frame.read( File "C:\ProgramData\Miniconda3\envs\helao\lib\site-packages\websockets\framing.py", line 105, in read data = await reader(2) File "C:\ProgramData\Miniconda3\envs\helao\lib\asyncio\streams.py", line 723, in readexactly await self._wait_for_data('readexactly') File "C:\ProgramData\Miniconda3\envs\helao\lib\asyncio\streams.py", line 517, in _wait_for_data await self._waiter File "C:\ProgramData\Miniconda3\envs\helao\lib\asyncio\selector_events.py", line 848, in _read_ready__data_received data = self._sock.recv(self.max_size) ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "E:\Matthias\GitHub\helao-dev\core\classes.py", line 92, in handle_socket async for message in websocket: File "C:\ProgramData\Miniconda3\envs\helao\lib\site-packages\websockets\protocol.py", line 439, in aiter yield await self.recv() File "C:\ProgramData\Miniconda3\envs\helao\lib\site-packages\websockets\protocol.py", line 509, in recv await self.ensure_open() File "C:\ProgramData\Miniconda3\envs\helao\lib\site-packages\websockets\protocol.py", line 803, in ensure_open raise self.connection_closed_exc() websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason Successfully terminated server 'potentiostat'. Successfully terminated server 'orchestrator'. All processes terminated. Removing 'pids_world.pck'

(helao) E:\Matthias\GitHub\helao-dev>

onepunchdan commented 3 years ago

Does this exception occur right after launch or after you press CTRL-x to terminate the process group?

onepunchdan commented 3 years ago

Thinking if it's the latter case, the orchestrator is being terminated after the action servers so websocket connections are left hanging. I'll modify helao.py's keyboard interrupt to first cancel the monitor_states() task before proceeding to terminate servers.

Yohko commented 3 years ago

Fixed by Dan with a13f4df