magland
commented
4 years ago First, could you make sure you are up to date with the current master branch? (although I don't expect that to solve this particular issue)
I wonder if this might be related to a connection timeout. Could you try to determine if the connection is timing out according to the following code in api.py:
async def incoming_message_handler(session, websocket):
async for message in websocket:
msg = json.loads(message)
session.handle_message(msg)
async def outgoing_message_handler(session, websocket):
while True:
try:
hi.wait(0)
except:
traceback.print_exc()
messages = session.check_for_outgoing_messages()
for message in messages:
await websocket.send(json.dumps(message))
if session.elapsed_sec_since_incoming_keepalive() > 60:
print('Closing session')
return
await asyncio.sleep(0.05)
hello @magland, we continue to have issues with websocket, sometimes connection immediately fails with log reported below. After restarting it works for some time, but then fails again
reported client info: {'type': 'reportClientInfo', 'clientInfo': {'feedUri': None, 'documentId': 'default', 'readOnly': False}} Process Process-7: Traceback (most recent call last): File "/usr/lib/python3.6/multiprocessing/process.py", line 258, in _bootstrap self.run() File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run self._target(*self._args, **self._kwargs) File "/labbox-ephys/python/labbox_ephys/api/_session.py", line 67, in _run_worker_session WS.handle_message(x['message']) File "/labbox-ephys/python/labbox_ephys/api/_workersession.py", line 55, in handle_message self._readonly = msg['clientInfo']['readonly'] KeyError: 'readonly' Connection closed. Error in connection handler Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/websockets/server.py", line 191, in handler await self.ws_handler(self, path) File "api.py", line 143, in connection_handler session.cleanup() File "/labbox-ephys/python/labbox_ephys/api/_session.py", line 18, in cleanup self._pipe_to_worker_process.send('exit') File "/usr/lib/python3.6/multiprocessing/connection.py", line 206, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/usr/lib/python3.6/multiprocessing/connection.py", line 404, in _send_bytes self._send(header + buf) File "/usr/lib/python3.6/multiprocessing/connection.py", line 368, in _send n = write(self._handle, buf) BrokenPipeError: [Errno 32] Broken pipe Task exception was never retrieved future: <Task finished coro=<outgoing_message_handler() done, defined at api.py:115> exception=EOFError()> Traceback (most recent call last): File "api.py", line 121, in outgoing_message_handler messages = session.check_for_outgoing_messages() File "/labbox-ephys/python/labbox_ephys/api/_session.py", line 23, in check_for_outgoing_messages msg = self._pipe_to_worker_process.recv() File "/usr/lib/python3.6/multiprocessing/connection.py", line 250, in recv buf = self._recv_bytes() File "/usr/lib/python3.6/multiprocessing/connection.py", line 407, in _recv_bytes buf = self._recv(4) File "/usr/lib/python3.6/multiprocessing/connection.py", line 383, in _recv raise EOFError EOFError Task exception was never retrieved future: <Task finished coro=<incoming_message_handler() done, defined at api.py:110> exception=ConnectionClosedError('code = 1006 (connection closed abnormally [internal]), no reason',)> Traceback (most recent call last): File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 827, in transfer_data message = await self.read_message() File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 895, in read_message frame = await self.read_data_frame(max_size=self.max_size) File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 971, in read_data_frame frame = await self.read_frame(max_size) File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 1051, in read_frame extensions=self.extensions, File "/usr/local/lib/python3.6/dist-packages/websockets/framing.py", line 105, in read data = await reader(2) File "/usr/lib/python3.6/asyncio/streams.py", line 674, in readexactly yield from self._wait_for_data('readexactly') File "/usr/lib/python3.6/asyncio/streams.py", line 464, in _wait_for_data yield from self._waiter concurrent.futures._base.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "api.py", line 111, in incoming_message_handler async for message in websocket: File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 439, in aiter yield await self.recv() File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 509, in recv await self.ensure_open() File "/usr/local/lib/python3.6/dist-packages/websockets/protocol.py", line 803, in ensure_open raise self.connection_closed_exc() websockets.exceptions.ConnectionClosedError: code = 1006 (connection closed abnormally [internal]), no reason