search

TimZaman / dotaclient

distributed RL spaghetti al arabiata
26 stars 7 forks source link

Make connection to dotaclient more resilient #1

Closed TimZaman closed 5 years ago

TimZaman commented 5 years ago 
kubectl logs dotaservice-deployment-f7ffdb9d5-fbk5l agent -f
2019-01-06 02:12:25,289 INFO     setup_model_cb(host=rmq.default.svc.cluster.local, port=5672)
2019-01-06 02:12:25,344 INFO     Received new model: version=224, size=1207326b
2019-01-06 02:12:25,347 INFO     Updated weights to version 224
2019-01-06 02:12:25,348 INFO     === Starting Episode 0.
2019-01-06 02:12:25,348 INFO     Starting game.
2019-01-06 02:12:25,349 ERROR    error on dispatch
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 573, in run_until_complete
    return future.result()
  File "agent.py", line 515, in main
    await game.play()
  File "agent.py", line 430, in play
    response = await asyncio.wait_for(self.dota_service.reset(self.config), timeout=120)
  File "/usr/lib/python3.7/asyncio/tasks.py", line 416, in wait_for
    return fut.result()
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 588, in __call__
    await stream.send_message(message, end=True)
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 163, in send_message
    await self.send_request()
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 132, in send_request
    protocol = await self._channel.__connect__()
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 477, in __connect__
    self._protocol = await self._create_connection()
  File "/root/.local/lib/python3.7/site-packages/grpclib/client.py", line 465, in _create_connection
    ssl=self._ssl)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 948, in create_connection
    raise exceptions[0]
  File "/usr/lib/python3.7/asyncio/base_events.py", line 935, in create_connection
    await self.sock_connect(sock, address)
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 475, in sock_connect
    return await fut
  File "/usr/lib/python3.7/asyncio/selector_events.py", line 505, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('127.0.0.1', 13337)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/.local/lib/python3.7/site-packages/aioamqp/protocol.py", line 333, in run
    yield from self.dispatch_frame()
  File "/root/.local/lib/python3.7/site-packages/aioamqp/protocol.py", line 280, in dispatch_frame
    frame = yield from self.get_frame()
  File "/root/.local/lib/python3.7/site-packages/aioamqp/protocol.py", line 264, in get_frame
    yield from frame.read_frame()
  File "/root/.local/lib/python3.7/site-packages/aioamqp/frame.py", line 453, in read_frame
    data = yield from self.reader.readexactly(7)
  File "/usr/lib/python3.7/asyncio/streams.py", line 679, in readexactly
    await self._wait_for_data('readexactly')
  File "/usr/lib/python3.7/asyncio/streams.py", line 473, in _wait_for_data
    await self._waiter
concurrent.futures._base.CancelledError
2019-01-06 02:12:55,431 INFO     Received new model: version=225, size=1207326b
2019-01-06 02:12:55,434 INFO     Updated weights to version 225
2019-01-06 02:13:40,232 INFO     Received new model: version=226, size=1207326b
2019-01-06 02:13:40,235 INFO     Updated weights to version 226
2019-01-06 02:14:24,366 INFO     Received new model: version=227, size=1207326b
2019-01-06 02:14:24,369 INFO     Updated weights to version 227
2019-01-06 02:15:05,102 INFO     Received new model: version=228, size=1207326b
2019-01-06 02:15:05,106 INFO     Updated weights to version 228
2019-01-06 02:15:45,595 INFO     Received new model: version=229, size=1207326b
2019-01-06 02:15:45,598 INFO     Updated weights to version 229
2019-01-06 02:16:23,744 INFO     Received new model: version=230, size=1207326b
2019-01-06 02:16:23,747 INFO     Updated weights to version 230
2019-01-06 02:16:59,806 INFO     Received new model: version=231, size=1207326b
2019-01-06 02:16:59,809 INFO     Updated weights to version 231
2019-01-06 02:17:40,834 INFO     Received new model: version=232, size=1207326b
2019-01-06 02:17:40,838 INFO     Updated weights to version 232
2019-01-06 02:18:19,436 INFO     Received new model: version=233, size=1207326b
2019-01-06 02:18:19,440 INFO     Updated weights to version 233
2019-01-06 02:19:18,664 INFO     Received new model: version=234, size=1207326b
2019-01-06 02:19:18,668 INFO     Updated weights to version 234
2019-01-06 02:20:00,478 INFO     Received new model: version=235, size=1207326b
2019-01-06 02:20:00,481 INFO     Updated weights to version 235
2019-01-06 02:20:40,009 INFO     Received new model: version=236, size=1207326b
2019-01-06 02:20:40,013 INFO     Updated weights to version 236
2019-01-06 02:21:20,128 INFO     Received new model: version=237, size=1207326b
2019-01-06 02:21:20,131 INFO     Updated weights to version 237
2019-01-06 02:22:01,556 INFO     Received new model: version=238, size=1207326b
2019-01-06 02:22:01,559 INFO     Updated weights to version 238
2019-01-06 02:22:49,645 INFO     Received new model: version=239, size=1207326b
2019-01-06 02:22:49,648 INFO     Updated weights to version 239
2019-01-06 02:23:26,087 INFO     Received new model: version=240, size=1207326b
2019-01-06 02:23:26,090 INFO     Updated weights to version 240
2019-01-06 02:24:05,643 INFO     Received new model: version=241, size=1207326b
2019-01-06 02:24:05,646 INFO     Updated weights to version 241
2019-01-06 02:24:54,556 INFO     Received new model: version=242, size=1207326b
2019-01-06 02:24:54,559 INFO     Updated weights to version 242
2019-01-06 02:25:29,720 INFO     Received new model: version=243, size=1207326b
2019-01-06 02:25:29,722 INFO     Updated weights to version 243
2019-01-06 02:26:10,312 INFO     Received new model: version=244, size=1207326b
2019-01-06 02:26:10,316 INFO     Updated weights to version 244
2019-01-06 02:26:47,232 INFO     Received new model: version=245, size=1207326b
2019-01-06 02:26:47,235 INFO     Updated weights to version 245
TimZaman commented 5 years ago

Probably same as #5 , closing.