AA freezes, in this case it stayed frozen for days. A proper crash would be better, so that we know when to restart it.
Giving up subgraph_query_count(...) after 2 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 192.168.64.14:7300 ssl:default [Connect call failed ('192.168.64.14', 7300)])
Giving up queries_per_second(...) after 3 tries (aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 192.168.64.14:7300 ssl:default [Connect call failed ('192.168.64.14', 7300)])
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 980, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore[return-value] # noqa
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
sock = await self._connect_sock(
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
return await fut
File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
TimeoutError: [Errno 110] Connect call failed ('192.168.64.14', 7300)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/autoagora/price_multiplier.py", line 111, in price_bandit_loop
queries_per_second = await environment.queries_per_second(
File "/usr/local/lib/python3.8/site-packages/backoff/_async.py", line 151, in retry
ret = await target(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/autoagora/subgraph_wrapper.py", line 45, in queries_per_second
query_count_1 = await subgraph_query_count(self.subgraph, metrics_endpoints)
File "/usr/local/lib/python3.8/site-packages/backoff/_async.py", line 151, in retry
ret = await target(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/autoagora/query_metrics.py", line 109, in subgraph_query_count
async with session.get(endpoint) as response:
File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 1141, in __aenter__
self._resp = await self._coro
File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 536, in _request
conn = await self._connector.connect(
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 540, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 901, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1206, in _create_direct_connection
raise last_exc
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1175, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 988, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 192.168.64.14:7300 ssl:default [Connect call failed ('192.168.64.14', 7300)]
AA freezes, in this case it stayed frozen for days. A proper crash would be better, so that we know when to restart it.