search

ualex73 / monitor_docker

Monitor Docker containers from Home Assistant
Apache License 2.0
267 stars 34 forks source link

Socket Proxy + HA issue #109

Closed tamimology closed 1 month ago

tamimology commented 1 year ago

I am running HA on docker, and have socket proxy as well, below is my compose for both:

  socket:
    container_name: socketproxy
    restart: $ALWAYS_ON_POLICY
    environment:
        PUID: $PUID
        PGID: $PGID
        LOG_LEVEL: err # debug,info,notice,warning,err,crit,alert,emerg
        # 0 to revoke access.
        # 1 to grant access.
        ## Granted by Default
        EVENTS: 1
        PING: 1
        VERSION: 1
        ## Revoked by Default
        # Security critical
        AUTH: 0
        SECRETS: 0
        POST: 1 # HA, Watchtower
        # Not always needed
        BUILD: 1 # HA
        COMMIT: 1 # HA
        CONFIGS: 1 # HA
        CONTAINERS: 1 # HA, Traefik, portainer, etc.
        DISTRIBUTION: 1 #HA
        EXEC: 1 # HA, ContainerWebTTy
        IMAGES: 1 # HA, Portainer
        INFO: 1 # HA, Portainer
        NETWORKS: 1 # HA, Portainer
        NODES: 1 # HA
        PLUGINS: 1 # HA
        SERVICES: 1 # HA, Portainer
        SESSION: 1 # HA
        SWARM: 1 # HA
        SYSTEM: 0
        TASKS: 1 # Portainer
        VOLUMES: 1 # Portainer
    volumes:
        - /var/run/docker.sock:/var/run/docker.sock
    ports:
        - 2375:2375
    network_mode: host
    privileged: true
    image: 'tecnativa/docker-socket-proxy:latest'

  homeassistant:
    container_name: homeassistant
    restart: $ALWAYS_ON_POLICY
    privileged: true
    environment:
      - DOCKER_HOST=192.168.1.10:2375 #also tried tcp://192.168.1.10:2375
      - PUID=$PUID
      - PGID=$PGID
      - TZ=$TZ
    volumes:
      - $LOCAL_TIME:/etc/localtime:ro
      - $PERSIST/homeassistant:/config:rw
    ports:
      - 8123:8123
    network_mode: host
    working_dir: /config
    depends_on:
       - mariadb
       - influxdb
    labels:
      - autoheal="true"
    image: 'ghcr.io/home-assistant/home-assistant:stable'

I did test the socket by running docker -H tcp://192.168.1.10:2375 ps on the host machine, and got the full detailed list of all containers I am running on the machine

And I have configured monitor_docker in HA as following:

monitor_docker: 
  - name: Docker
    url: tcp://192.168.1.10:2375 #tcp://172.18.0.199:2375
    monitored_conditions:
      - version
      - containers_total
      - containers_running
      - containers_paused
      - containers_stopped
      - status
    containers:
      - homeassistant

When I boot HA, in the logs, I keep getting the below (this comes out for every single container I have, even if i did not include in the configuration to be monitored):

29/09/2022 16:31:01
2022-09-29 16:31:01.232 ERROR (Thread-2 (RunDocker)) [custom_components.monitor_docker.helpers] [Docker] letsencrypt: Container not available anymore (3) (DockerError(900, 'Cannot connect to Docker Engine via tcp://192.168.1.10:2375 [Server disconnected]'))
29/09/2022 16:31:01
Traceback (most recent call last):
29/09/2022 16:31:01
  File "/usr/local/lib/python3.10/site-packages/aiodocker/docker.py", line 248, in _do_query
29/09/2022 16:31:01
    response = await self.session.request(
29/09/2022 16:31:01
  File "/usr/local/lib/python3.10/site-packages/aiohttp/streams.py", line 616, in read
29/09/2022 16:31:01
    await self._waiter
29/09/2022 16:31:01
aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected
29/09/2022 16:31:01
29/09/2022 16:31:01
During handling of the above exception, another exception occurred:
29/09/2022 16:31:01
29/09/2022 16:31:01
  File "/usr/local/lib/python3.10/site-packages/aiodocker/docker.py", line 300, in _query_json
29/09/2022 16:31:01
    async with self._query(
29/09/2022 16:31:01
  File "/usr/local/lib/python3.10/site-packages/aiodocker/utils.py", line 309, in __aenter__
29/09/2022 16:31:01
    resp = await self._coro
29/09/2022 16:31:01
  File "/usr/local/lib/python3.10/site-packages/aiodocker/docker.py", line 261, in _do_query
29/09/2022 16:31:01
aiodocker.exceptions.DockerError: DockerError(900, 'Cannot connect to Docker Engine via tcp://192.168.1.10:2375 [Server disconnected]')
mydliar commented 1 year ago

I'm running Monitor Docker against the docker-socket-proxy running alongside the actual HA, in the same Docker on Synology NAS. The reason why I'm using docker-socket-proxy is that it's not possible to mount /var/run/docker.sock in Synology Docker GUI. When installed and configured, all worked fine. Now checking the container statuses in HA, and HA logs, I can see the same errors as reported above:

2022-11-02 08:28:31.904 ERROR (Thread-3 (RunDocker)) [custom_components.monitor_docker.helpers] [Docker]: run_docker_info (DockerError(900, 'Cannot connect to Docker Engine via tcp://localhost:2375 [Server disconnected]')) Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/aiodocker/docker.py", line 248, in _do_query response = await self.session.request( File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 559, in _request await resp.start(conn) File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 898, in start message, payload = await protocol.read() # type: ignore[union-attr] File "/usr/local/lib/python3.10/site-packages/aiohttp/streams.py", line 616, in read await self._waiter aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/config/custom_components/monitor_docker/helpers.py", line 489, in _run_docker_info info = await self._api.system.info() File "/usr/local/lib/python3.10/site-packages/aiodocker/system.py", line 16, in info response = await self.docker._query_json("info", method="GET") File "/usr/local/lib/python3.10/site-packages/aiodocker/docker.py", line 300, in _query_json async with self._query( File "/usr/local/lib/python3.10/site-packages/aiodocker/utils.py", line 309, in aenter resp = await self._coro File "/usr/local/lib/python3.10/site-packages/aiodocker/docker.py", line 261, in _do_query raise DockerError( aiodocker.exceptions.DockerError: DockerError(900, 'Cannot connect to Docker Engine via tcp://localhost:2375 [Server disconnected]')

The dockerproxy container log doesn't show anything unusual, only lots of usual monitoring activities, such as: 172.17.0.1:48954 [02/Nov/2022:08:45:49.715] dockerfrontend dockerbackend/dockersocket 0/0/0/1048/1048 200 2240 - - ---- 3/3/1/1/0 0/0 "GET /v1.41/containers/518a8ff77ec9695cd12831726343178d07843dbfb988e615fedfdd59fa80d2be/stats?stream=0 HTTP/1.1"

The docker entities in HA seem to recover every time after a few minutes. Some monitored conditions seem to be affected more affected than others:

image

image

image

The only three monitored conditions which are not affected by this are version, containers_running, containers_total. All conditions recover to normal state after some time, without any intervention: 2022-11-02 09:22:06.583 INFO (MainThread) [custom_components.monitor_docker.sensor] [Docker] appdeamon: Initializing sensor with parameter: Health

quinten94b commented 1 year ago

I'm running Monitor Docker against the docker-socket-proxy running alongside the actual HA, in the same Docker on Synology NAS. The reason why I'm using docker-socket-proxy is that it's not possible to mount /var/run/docker.sock in Synology Docker GUI. When installed and configured, all worked fine. Now checking the container statuses in HA, and HA logs, I can see the same errors as reported above:

2022-11-02 08:28:31.904 ERROR (Thread-3 (RunDocker)) [custom_components.monitor_docker.helpers] [Docker]: run_docker_info (DockerError(900, 'Cannot connect to Docker Engine via tcp://localhost:2375 [Server disconnected]')) Traceback (most recent call last): File "/usr/local/lib/python3.10/site-packages/aiodocker/docker.py", line 248, in _do_query response = await self.session.request( File "/usr/local/lib/python3.10/site-packages/aiohttp/client.py", line 559, in _request await resp.start(conn) File "/usr/local/lib/python3.10/site-packages/aiohttp/client_reqrep.py", line 898, in start message, payload = await protocol.read() # type: ignore[union-attr] File "/usr/local/lib/python3.10/site-packages/aiohttp/streams.py", line 616, in read await self._waiter aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/config/custom_components/monitor_docker/helpers.py", line 489, in _run_docker_info info = await self._api.system.info() File "/usr/local/lib/python3.10/site-packages/aiodocker/system.py", line 16, in info response = await self.docker._query_json("info", method="GET") File "/usr/local/lib/python3.10/site-packages/aiodocker/docker.py", line 300, in _query_json async with self._query( File "/usr/local/lib/python3.10/site-packages/aiodocker/utils.py", line 309, in aenter resp = await self._coro File "/usr/local/lib/python3.10/site-packages/aiodocker/docker.py", line 261, in _do_query raise DockerError( aiodocker.exceptions.DockerError: DockerError(900, 'Cannot connect to Docker Engine via tcp://localhost:2375 [Server disconnected]')

The dockerproxy container log doesn't show anything unusual, only lots of usual monitoring activities, such as: 172.17.0.1:48954 [02/Nov/2022:08:45:49.715] dockerfrontend dockerbackend/dockersocket 0/0/0/1048/1048 200 2240 - - ---- 3/3/1/1/0 0/0 "GET /v1.41/containers/518a8ff77ec9695cd12831726343178d07843dbfb988e615fedfdd59fa80d2be/stats?stream=0 HTTP/1.1"

The docker entities in HA seem to recover every time after a few minutes. Some monitored conditions seem to be affected more affected than others:

image

image

image

The only three monitored conditions which are not affected by this are version, containers_running, containers_total. All conditions recover to normal state after some time, without any intervention: 2022-11-02 09:22:06.583 INFO (MainThread) [custom_components.monitor_docker.sensor] [Docker] appdeamon: Initializing sensor with parameter: Health

Did you find a solution?

From my HA vm to Synology dockers this docker is working. But when I run this docker on my unraid server, I have the same issues as you.

ualex73 commented 1 month ago

This issue still persists?

ualex73 commented 1 month ago

I will close this one, if it happens again, please raise a new issue.