Dotaclient deadlocks on rmq connection loss

Agent:

$ tzaman@Tims-Mac-Pro dotaclient (master) $ kubectl logs dotaservice-deployment-8c499c96-2mgbd agent
2019-01-06 21:18:41,153 INFO     setup_model_cb(host=rmq.default.svc.cluster.local, port=5672)
2019-01-06 21:18:41,283 INFO     Received new model: version=284, size=1207326b
2019-01-06 21:18:41,287 INFO     Updated weights to version 284
2019-01-06 21:18:41,288 INFO     === Starting Episode 0.
2019-01-06 21:18:41,288 INFO     Starting game.
2019-01-06 21:18:41,290 ERROR    error on dispatch
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "agent.py", line 515, in main
    await game.play()
  File "agent.py", line 430, in play
    response = await asyncio.wait_for(self.dota_service.reset(self.config), timeout=120)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 588, in __call__
    await stream.send_message(message, end=True)
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 163, in send_message
    await self.send_request()
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 132, in send_request
    protocol = await self._channel.__connect__()
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 477, in __connect__
    self._protocol = await self._create_connection()
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 465, in _create_connection
    ssl=self._ssl)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 948, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.7/asyncio/base_events.py", line 935, in create_connection
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 475, in sock_connect
    return await fut
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 505, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 13337)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.7/site-packages/aioamqp/protocol.py", line 333, in run
    yield from self.dispatch_frame()
  File "/root/.local/lib/python3.7/site-packages/aioamqp/protocol.py", line 288, in dispatch_frame
    yield from channel.dispatch_frame(frame)
  File "/root/.local/lib/python3.7/site-packages/aioamqp/channel.py", line 111, in dispatch_frame
    yield from methods[(frame.class_id, frame.method_id)](frame)
  File "/root/.local/lib/python3.7/site-packages/aioamqp/channel.py", line 631, in basic_deliver
    content_body_frame = yield from self.protocol.get_frame()
  File "/root/.local/lib/python3.7/site-packages/aioamqp/protocol.py", line 264, in get_frame
    yield from frame.read_frame()
  File "/root/.local/lib/python3.7/site-packages/aioamqp/frame.py", line 462, in read_frame
    payload_data = yield from self.reader.readexactly(self.frame_length)
  File "/usr/lib/python3.7/asyncio/streams.py", line 679, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.7/asyncio/streams.py", line 473, in _wait_for_data
    await self._waiter
concurrent.futures._base.CancelledError

Dotaservice (0.3.2):

$ tzaman@Tims-Mac-Pro dotaclient (master) $ kubectl logs dotaservice-deployment-8c499c96-2mgbd dotaservice
2019-01-06 21:18:41,932 INFO     DotaService 0.3.2 serving on :13337

TimZaman / dotaclient

Dotaclient deadlocks on rmq connection loss #5