search

project-receptor / python-receptor

Project Receptor is a flexible multi-service relayer with remote execution and orchestration capabilities linking controllers with executors across a mesh of nodes.
Other
32 stars 21 forks source link

[HIGH] - When a node disconnects - bad things occur on controller #66

Closed psav closed 4 years ago

psav commented 4 years ago

The pings had finished, looks like something happens to the Transport to make it None

DEBUG 2019-12-10 08:37:24,905 controller router Forwarding frame 296341455170759310604991198164716191918 to ping_node
DEBUG 2019-12-10 08:37:25,234 controller base waiting for HI
DEBUG 2019-12-10 08:37:25,235 controller base sending HI
DEBUG 2019-12-10 08:37:25,235 controller base sending routes
DEBUG 2019-12-10 08:37:25,235 controller receptor Emitting Route Advertisements, excluding set()
ERROR 2019-12-10 08:37:25,247 controller base watch_queue: error received trying to write
Traceback (most recent call last):
  File "/home/psavage/workspace/receptor/receptor/connection/base.py", line 39, in watch_queue
    await conn.send(msg)
  File "/home/psavage/workspace/receptor/receptor/connection/sock.py", line 31, in send
    await self.writer.drain()
  File "/usr/lib64/python3.7/asyncio/streams.py", line 348, in drain
    await self._protocol._drain_helper()
  File "/usr/lib64/python3.7/asyncio/streams.py", line 202, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost
INFO 2019-12-10 08:37:25,249 controller receptor Removing connection <receptor.connection.sock.RawSocket object at 0x7fa324363d50> for node ping_node
Task exception was never retrieved
future: <Task finished coro=<serve() done, defined at /home/psavage/workspace/receptor/receptor/connection/sock.py:51> exception=TypeError("object NoneType can't be used in 'await' expression")>
Traceback (most recent call last):
  File "/home/psavage/workspace/receptor/receptor/connection/base.py", line 39, in watch_queue
    await conn.send(msg)
  File "/home/psavage/workspace/receptor/receptor/connection/sock.py", line 31, in send
    await self.writer.drain()
  File "/usr/lib64/python3.7/asyncio/streams.py", line 348, in drain
    await self._protocol._drain_helper()
  File "/usr/lib64/python3.7/asyncio/streams.py", line 202, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/psavage/workspace/receptor/receptor/connection/sock.py", line 53, in serve
    await factory().server(t)
  File "/home/psavage/workspace/receptor/receptor/connection/base.py", line 123, in server
    await self.start_processing()
  File "/home/psavage/workspace/receptor/receptor/connection/base.py", line 98, in start_processing
    return await self.write_task
  File "/home/psavage/workspace/receptor/receptor/connection/base.py", line 43, in watch_queue
    return await conn.close()
TypeError: object NoneType can't be used in 'await' expression
DEBUG 2019-12-10 08:37:25,251 controller receptor Emitting Route Advertisements, excluding set()
ERROR 2019-12-10 08:37:25,251 controller receptor message_handler
Traceback (most recent call last):
  File "/home/psavage/workspace/receptor/receptor/receptor.py", line 93, in message_handler
    data = await buf.get()
  File "/home/psavage/workspace/receptor/receptor/messages/envelope.py", line 157, in get
    return await self.q.get()
  File "/usr/lib64/python3.7/asyncio/queues.py", line 159, in get
    await getter
concurrent.futures._base.CancelledError
DEBUG 2019-12-10 08:37:25,258 controller base starting normal loop