search

victronenergy / venus-influx-loader

NodeJS server that takes from MQTT into Influx, and config UI and still more
MIT License
14 stars 6 forks source link

bug: InfluxDB auto reconnect is broken #2

Closed mman closed 9 months ago

mman commented 11 months ago

I am running venus-influx-loader on Raspberry PI talking to Influx DB via basic docker-compose.yaml file that uses (old and now deprecated) dependency mechanism to start containers in order like this:

services:
  influxdb:
    restart: always

  loader:
    restart: always
    depends_on:
      influxdb:
        condition: service_healthy

This works great during manual docker compose up assuming service healthcheck is properly implemented (full details below).

But this fails miserably during automated system reboot because docker will ignore depends_on condition after reboot.

After experiencing multiple times that my loader does not work after automatic system reboot, I noticed that loader should attempt to automatically reconnect in 5 second intervals should InfluxDB become unreachable, which would work around the wrong startup order docker problem. But the code does not work. After the system reboot, loader service attempts to connect once and then never again:

venus-influx-loader: Use --help to learn how to use this program
venus-influx-loader: Config Path: /config
venus-influx-loader: Discovery API: disabled
venus-influx-loader: Admin API: /admin-api/
venus-influx-loader: Grafana JSON Datasource API: /grafana-api/
venus-influx-loader: API Port: 8088
[info] [influxdb] Attempting connection to influxdb:8086/venus using undefined:***** 
[info] [upnp] Running UPNP Discovery... 
[info] [venus-influx-loader] setting up /admin-api/ routes 
[info] [venus-influx-loader] setting up /grafana-api/ routes 
[info] [venus-influx-loader] running at 0.0.0.0:8088 
[error] [influxdb] Unable to connect: Error: connect ECONNREFUSED 172.20.0.2:8086

The code managing this should be here:

https://github.com/victronenergy/venus-influx-loader/blob/536c20726786788bade1defb536e34fe4fc7b190/src/server/influxdb.js#L47-L60

TODO: Need to investigate this and make sure venus-influx-loader properly handles InfluxDB not being available initially, or disappearing and then re-appearing on the network to make the functionality robust.

Full docker-compose.yaml for reference:

version: '2.4'
services:
  loader:
    restart: always
    image: "sodaengine/venus-influx-loader:main"
    ports:
      - "8088:8088"
    volumes:
      - type: bind
        source: /mnt/lacie/venus-docker-grafana/config
        target: /config
    depends_on:
      influxdb:
        condition: service_healthy
  influxdb:
    restart: always
    image: "influxdb:1.8"
    ports:
      - "8086:8086"
    volumes:
      - type: bind
        source: /mnt/lacie/venus-docker-grafana/influxdb
        target: /var/lib/influxdb
    environment:
      - INFLUXDB_HTTP_LOG_ENABLED=false
    healthcheck:
      test: "curl -f http://influxdb:8086/ping"
      interval: 5s
      timeout: 10s
  grafana:
    restart: always
    image: "sodaengine/venus-grafana:main"
    ports:
      - "3000:3000"
    environment:
      - GF_SERVER_DOMAIN=nemo.sodaengine.com
      - GF_SERVER_ROOT_URL=https://nemo.sodaengine.com/grafana/
      - GF_SERVER_SERVE_FROM_SUB_PATH=true
      - VIL_INFLUXDB_URL=http://influxdb:8086
      - VIL_GRAFANA_API_URL=http://loader:8088/grafana-api
    depends_on:
      influxdb:
        condition: service_healthy
      loader:
        condition: service_healthy
mman commented 9 months ago

Closing via https://github.com/victronenergy/venus-influx-loader/pull/16.