ualex73 / monitor_docker

Monitor Docker containers from Home Assistant
Apache License 2.0
267 stars 34 forks source link

Started getting weird errors recently #57

Closed Dinth closed 1 month ago

Dinth commented 3 years ago
2021-03-10 18:00:17 ERROR (Thread-3) [root] Uncaught thread exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
raise exceptions[0]
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
sock = await self._connect_sock(
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
await self.sock_connect(sock, address)
File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
return await fut
File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
raise OSError(err, f'Connect call failed {address}')
ConnectionRefusedError: [Errno 111] Connect call failed ('10.10.1.12', 2375)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/aiodocker/docker.py", line 241, in _do_query
response = await self.session.request(
File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 520, in _request
conn = await self._connector.connect(
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 892, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1051, in _create_direct_connection
raise last_exc
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 1020, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
File "/usr/local/lib/python3.8/site-packages/aiohttp/connector.py", line 975, in _wrap_create_connection
raise client_error(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host 10.10.1.12:2375 ssl:default [Connect call failed ('10.10.1.12', 2375)]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "/config/custom_components/monitor_docker/__init__.py", line 82, in RunDocker
hass.data[DOMAIN][entry[CONF_NAME]][API] = DockerAPI(hass, entry)
File "/config/custom_components/monitor_docker/helpers.py", line 146, in __init__
version = self._loop.run_until_complete(self._api.version())
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/aiodocker/docker.py", line 166, in version
data = await self._query_json("version")
File "/usr/local/lib/python3.8/site-packages/aiodocker/docker.py", line 293, in _query_json
async with self._query(
File "/usr/local/lib/python3.8/site-packages/aiodocker/utils.py", line 309, in __aenter__
resp = await self._coro
File "/usr/local/lib/python3.8/site-packages/aiodocker/docker.py", line 233, in _do_query
await self._check_version()
File "/usr/local/lib/python3.8/site-packages/aiodocker/docker.py", line 185, in _check_version
ver = await self._query_json("version", versioned_api=False)
File "/usr/local/lib/python3.8/site-packages/aiodocker/docker.py", line 293, in _query_json
async with self._query(
File "/usr/local/lib/python3.8/site-packages/aiodocker/utils.py", line 309, in __aenter__
resp = await self._coro
File "/usr/local/lib/python3.8/site-packages/aiodocker/docker.py", line 254, in _do_query
raise DockerError(
aiodocker.exceptions.DockerError: DockerError(900, "Cannot connect to Docker Engine via tcp://10.10.1.12:2375 [Cannot connect to host 10.10.1.12:2375 ssl:default [Connect call failed ('10.10.1.12', 2375)]]")
ualex73 commented 3 years ago

This looks there is no connectivity to the Docker API on 10.10.1.12 on port 2375?

redpizza69 commented 3 years ago

I am to getting the same "Cannot connect" messages one by one for each of my containers. I am using a docker proxy (tecnativa/docker-socket-proxy) rather than docker.sock. Once a container get this error then I get no more stats for that container, so gradually over time all my containers become 'lost'.

Is it possible that due to the relative high polling of the proxy (I am polling around 20 containers) that occasionally a connection times out, but then monitor_docker doesn't try that container again?

IDmedia commented 3 years ago

I also get disconnect warnings when setting the scan_interval to a value lower then 30. Not sure yet if it's the proxy not able to handle the number of requests at that speed or if Home Assistant struggeles with processing them fast enough. Both HA and the proxy are on powerful hardware, so a bit strange... I'm running 34 docker containers at the moment.

ualex73 commented 2 years ago

This issue is resolved in v1.11?

ualex73 commented 1 month ago

I will close this old issue. If still persists, please open a new issue.