search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.93k stars 5.6k forks source link

MQTT connection failures with Mosquitto #7648

Closed tkerby closed 3 years ago

tkerby commented 4 years ago

I'm seeing issues with communication between telegraf and mosquito in a docker setup where the connection fails and does not recover. This may be related to issue #4594 as there is similar behaviour although I believe that was resolved.

Relevant telegraf.conf:

# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
  name_override = "mqtt_home"

  ## MQTT broker URLs to be used. The format should be scheme://host:port,
  ## schema can be tcp, ssl, or ws.
  servers = ["tcp://mosquitto:1883"]

  ## Topics that will be subscribed to.
  topics = ["home/devices/+/+/up"]

  ## The message topic will be stored in a tag specified by this value.  If set
  ## to the empty string no topic tag will be created.
  # topic_tag = "topic"

  ## QoS policy for messages
  ##   0 = at most once
  ##   1 = at least once
  ##   2 = exactly once
  ##
  ## When using a QoS of 1 or 2, you should enable persistent_session to allow
  ## resuming unacknowledged messages.
  qos = 0

  ## Connection timeout for initial connection in seconds
  connection_timeout = "60s"

  ## Maximum messages to read from the broker that have not been written by an
  ## output.  For best throughput set based on the number of metrics within
  ## each message and the size of the output's metric_batch_size.
  ##
  ## For example, if each message from the queue contains 10 metrics and the
  ## output metric_batch_size is 1000, setting this to 100 will ensure that a
  ## full batch is collected and the write is triggered immediately without
  ## waiting until the next flush_interval.
  max_undelivered_messages = 1

  ## Persistent session disables clearing of the client session on connection.
  ## In order for this option to work you must also set client_id to identity
  ## the client.  To receive messages that arrived while the client is offline,
  ## also set the qos option to 1 or 2 and don't forget to also set the QoS when
  ## publishing.
  persistent_session = false

  ## If unset, a random client ID will be generated.
  client_id = "telegraf1"

  ## Username and password to connect MQTT server.
  username = "*******"
  password = "*******"

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "json"
  interval = "20s"

System info:

Telegraf 1.14.3 with mosquito 1.6.10

Docker

version: '3.8'
services:
    node-red:
        container_name: node-red
        restart: unless-stopped
        image: 'nodered/node-red:latest'
        environment:
            - TZ=Europe/London
        ports:
            - '1880:1880'
        networks:
            - iot-net
        volumes:
            - /mnt/SSD/docker/node-red/data:/data
        user: root:root
    mosquitto:
        container_name: mosquitto
        restart: unless-stopped
        image: eclipse-mosquitto
        environment:
            - TZ=Europe/London
        ports:
            - '1883:1883'
            - '9001:9001'
        networks:
            - iot-net
        volumes:
            - /mnt/SSD/docker/mosquitto/config:/mosquitto/config
            - /mnt/SSD/docker/mosquitto/data:/mosquitto/data
            - /mnt/SSD/docker/mosquitto/log:/mosquitto/log
    influxdb:
        container_name: influxdb
        restart: unless-stopped
        image: influxdb
        environment:
            - TZ=Europe/London
            - INFLUXDB_HTTP_AUTH_ENABLED=true
        networks:
            - iot-net
        volumes:
            - /mnt/SSD/docker/influxdb/data:/var/lib/influxdb
        ports:
            # The API for InfluxDB is served on port 8086
            - "8082:8082"
            - "8086:8086"
            # UDP Port
            - "8089:8089"
        privileged: true
    telegraf:
        container_name: telegraf
        restart: unless-stopped
        image: telegraf
        networks:
            - iot-net
        environment:
            - interval=60s
            - flush_interval=60s
        depends_on:
            - influxdb
            - mosquitto
        privileged: true
        ports:
            - 8125:8125
            - 8092:8092
            - 8094:8094
        volumes:
            - /mnt/SSD/docker/telegraf/config:/etc/telegraf
            - /var/run/docker.sock:/var/run/docker.sock
    grafana:
        container_name: grafana
        restart: unless-stopped
        image: grafana/grafana
        environment:
            - TZ=Europe/London
        depends_on:
            - influxdb
        networks:
            - iot-net
        volumes:
            - /mnt/SSD/docker/grafana/data:/var/lib/grafana
            - /mnt/SSD/docker/grafana/config/grafana.ini:/etc/grafana/grafana.ini      
        ports:
            - '3000:3000'
    nginx:
        container_name: nginx
        restart: unless-stopped
        image: nginx
        depends_on:
            - grafana
            - node-red
        environment:
            - TZ=Europe/London
        ports:
            - '80:80'
            - '443:443'
        networks:
            - iot-net
        volumes:
            - /mnt/SSD/docker/nginx/config:/etc/nginx
            - /mnt/SSD/docker/nginx/certs:/etc/ssl/private

networks:
    iot-net: null

Steps to reproduce:

Mosquitto setup with basic password authentication. Connected ok for the first 10 hours or so then failed. No config changes at the time. Fails on restart even when cleaned with docker-compose down.

I'm pulling data from the local mosquito broker.

Problem seems to be related to pingresp - see an error on both sides

Expected behavior:

Expect to see Telegraf pulling the mqqt data into influxdb. Sensor data is approx every 15 seconds

Actual behavior:

No data transferred to influxdb. Repeated reconnects

Additional info:

Can subscribe to the topic from a desktop without an issue: C:\Program Files\mosquitto>mosquitto_sub -h grafana.local -t home/devices/+/+/up -u -P -v home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.4375} home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}

I'm also ok pulling mqtt data from two other remote sources (Things network).

I've tried

Mosquitto log:

2020-06-08T12:26:48: mosquitto version 1.6.10 starting
2020-06-08T12:26:48: Config loaded from /mosquitto/config/mosquitto.conf.
2020-06-08T12:26:48: Opening ipv4 listen socket on port 1883.
2020-06-08T12:26:48: Opening ipv6 listen socket on port 1883.
2020-06-08T12:26:51: New connection from 192.168.1.238 on port 1883.
2020-06-08T12:26:51: New client connected from 192.168.1.238 as beerfridge (p2, c1, k15, u'sensors').
2020-06-08T12:26:53: New connection from 172.18.0.6 on port 1883.
2020-06-08T12:26:53: New client connected from 172.18.0.6 as telegraf1 (p2, c1, k60, u'sensors').
2020-06-08T12:28:13: Socket error on client telegraf1, disconnecting.
2020-06-08T12:28:20: New connection from 172.18.0.6 on port 1883.
2020-06-08T12:28:20: New client connected from 172.18.0.6 as telegraf1 (p2, c1, k60, u'sensors').
2020-06-08T12:29:35: Socket error on client telegraf1, disconnecting.

Telegraf log:

2020-06-08T12:26:53Z I! Starting Telegraf 1.14.3
2020-06-08T12:26:53Z I! Using config file: /etc/telegraf/telegraf.conf
2020-06-08T12:26:53Z I! Loaded inputs: cpu processes swap docker openweathermap disk diskio kernel mem system mqtt_consumer mqtt_consumer mqtt_consumer
2020-06-08T12:26:53Z I! Loaded aggregators:
2020-06-08T12:26:53Z I! Loaded processors: regex
2020-06-08T12:26:53Z I! Loaded outputs: influxdb
2020-06-08T12:26:53Z I! Tags enabled: host=57ce4e91341b
2020-06-08T12:26:53Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"57ce4e91341b", Flush Interval:1m0s
2020-06-08T12:26:53Z I! [inputs.mqtt_consumer] Connected [tcp://eu.thethings.network:1883]
2020-06-08T12:26:53Z I! [inputs.mqtt_consumer] Connected [tcp://eu.thethings.network:1883]
2020-06-08T12:26:53Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2020-06-08T12:28:13Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
2020-06-08T12:28:20Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2020-06-08T12:29:35Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
2020-06-08T12:29:40Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2020-06-08T12:31:00Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
2020-06-08T12:31:20Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2020-06-08T12:32:40Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
2020-06-08T12:33:00Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2020-06-08T12:34:20Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
2020-06-08T12:34:40Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2020-06-08T12:36:00Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
2020-06-08T12:36:20Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2020-06-08T12:37:40Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
2020-06-08T12:38:00Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
2020-06-08T12:39:20Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
danielnelson commented 4 years ago

I'm not sure what the issue is here, but I found this issue that seems like it might be related: https://github.com/eclipse/paho.mqtt.golang/issues/263#issuecomment-444819757. The issue opened recreated the docker network and the error was no longer reproducable.

Perhaps you could collect a packet capture of both Telegraf and mosquitto_sub:

sudo tcpdump -s 0 -w telegraf-mqtt.pcap host 127.0.0.1 and port 1883
tkerby commented 4 years ago

I’ve also tried VerneMQ as the mqtt broker tonight and see the same issue.

In terms of networks, the network is destroyed and recreated as part of the docker-compose down/up commands so doesn’t seem to be that.

I’m thinking it may have to do with whether mosquitto is fully available when docker starts Telegraf. I’ve setup the dependency but it may not have sufficient time. That could explain why the external things network mqtt servers don’t show an issue.

For the network log, what container should i run in or should I try and connect to the IoT-net externally?

danielnelson commented 4 years ago

Try to collect the packet capture from the same container that runs Telegraf.

tkerby commented 4 years ago

Got the capture and the ping requests and responses look ok. I've not included the full file here as there are some passwords etc but the screenshot should be enough - let me know if you want anything else pulled.

Interestingly, it seems after the ping response, one more message (an mqtt pub) is sent before the reconnect happens. I'm running a longer capture to try and prove this

image

tkerby commented 4 years ago

I can confirm that there is a Publish Message after every Ping Response before the reconnect happens. I suspect telegraf doesnt act on the Ping Response and drops its connection, mosquitto then sees the dropped connection when it tries to publish thus generating the socket error.

For reference, I built tcpdump for arm (this is on a Raspberry Pi 4) using the following Dockerfile

FROM alpine
RUN apk add --update --no-cache tcpdump && rm -rf /var/cache/apk/*
VOLUME  [ "/tcpdump" ]
ENTRYPOINT [ "/usr/sbin/tcpdump" ]
CMD [ "-C", "1000", "-v", "-i", "any", "-w", "/tcpdump/tcpdump.pcap" ]

You can then connect to the container and dump a capture file with:

docker run --rm --net=container:telegraf -v $PWD/tcpdump2:/tcpdump --detach --name tcpdump tcpdump:1.0

Don't connect with --net=host or you see lots of spurious tcp transmissions as you log every communication twice!

tkerby commented 4 years ago

Also now removed other mqtt input plugins to see if there was any conflict - the same problem is visible even when this is the only mqtt input

tkerby commented 4 years ago

Dropped into the container and ran telegraf --test-wait 120 to see if data was being pulled ok. It seems it connects fine without pingresp problems and I get

mqtt_home,host=5950ed9f4c07,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=4.4375 1591784544681450142 mqtt_home,host=5950ed9f4c07,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=4.4375 1591784559680842194 mqtt_home,host=5950ed9f4c07,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=4.4375 1591784574683436644 mqtt_home,host=5950ed9f4c07,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=4.4375 1591784589682337592 mqtt_home,host=5950ed9f4c07,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=4.4375 1591784604682882968 mqtt_home,host=5950ed9f4c07,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=4.4375 1591784619684979054 mqtt_home,host=5950ed9f4c07,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=4.4375 1591784634685235101 mqtt_home,host=5950ed9f4c07,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=4.4375 1591784649686340667

danielnelson commented 4 years ago

I'm I understanding correctly that --test-wait is working but not normal operation?

tkerby commented 4 years ago

@danielnelson Seems to be the case although I'd caveat that I've not tested it robustly

I did look at whether it could be ticker related - e.g. wait for the pingresp but the connection isn't made for 20s to confirm. I changed the timing down to a few seconds for that plugin but no difference.

Any thoughts?

tkerby commented 4 years ago

@danielnelson Can absolutely confirm that all is ok in --test-wait but not in normal operation

I added the " command: --test-wait 600" flag to the docker compose file to ensure only one copy of telegraf was running and it ran for sufficient time. No errors on the mosquito side and I could see telegraf happily receiving data from all mqtt servers.

danielnelson commented 4 years ago

With --test-wait does it still show this log message at the beginning before it prints out the messages?

Error in plugin: connection lost: pingresp not received, disconnecting
tkerby commented 4 years ago

@danielnelson I dont see that message with --test-wait

Logs of all containers below. For reference, the disconnect on the mosquito device beer fridge was intentional to see if it recovered.


pi@grafana:~/docker $ docker-compose up
Starting node-red  ... done
Starting influxdb  ... done
Starting mosquitto ... done
Starting telegraf  ... done
Starting grafana   ... done
Starting nginx     ... done
Attaching to mosquitto, node-red, influxdb, grafana, telegraf, nginx
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Starting Grafana" logger=server version=7.0.3 commit=00ee734baf branch=HEAD compiled=2020-06-03T16:19:31+0100
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.logs=/var/log/grafana"
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.plugins=/var/lib/grafana/plugins"
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.provisioning=/etc/grafana/provisioning"
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config overridden from command line" logger=settings arg="default.log.mode=console"
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana"
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana"
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins"
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="App mode production" logger=settings
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3
mosquitto    | 2020-06-11T15:31:35: mosquitto version 1.6.10 starting
mosquitto    | 2020-06-11T15:31:35: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto    | 2020-06-11T15:31:35: Opening ipv4 listen socket on port 1883.
mosquitto    | 2020-06-11T15:31:35: Opening ipv6 listen socket on port 1883.
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Starting DB migration" logger=migrator
mosquitto    | 2020-06-11T15:31:37: New connection from 192.168.1.239 on port 1883.
mosquitto    | 2020-06-11T15:31:37: New client connected from 192.168.1.239 as beerfridge (p2, c1, k15, u'sensors').
mosquitto    | 2020-06-11T15:31:38: New connection from 172.30.0.5 on port 1883.
mosquitto    | 2020-06-11T15:31:38: New client connected from 172.30.0.5 as Telegraf-Consumer-oAzAu (p2, c1, k60, u'sensors').
influxdb     | ts=2020-06-11T15:31:35.736833Z lvl=info msg="InfluxDB starting" log_id=0NK~yfj0000 version=1.8.0 branch=1.8 commit=781490de48220d7695a05c29e5a36f550a4568f5
influxdb     | ts=2020-06-11T15:31:35.736887Z lvl=info msg="Go runtime" log_id=0NK~yfj0000 version=go1.13.8 maxprocs=4
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Starting plugin search" logger=plugins
grafana      | t=2020-06-11T16:31:38+0100 lvl=info msg="Registering plugin" logger=plugins name="Direct Input"
nginx        | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
influxdb     | ts=2020-06-11T15:31:35.852560Z lvl=info msg="Using data dir" log_id=0NK~yfj0000 service=store path=/var/lib/influxdb/data
influxdb     | ts=2020-06-11T15:31:35.857541Z lvl=info msg="Compaction settings" log_id=0NK~yfj0000 service=store max_concurrent_compactions=2 throughput_bytes_per_second=50331648 throughput_bytes_per_second_burst=50331648
influxdb     | ts=2020-06-11T15:31:35.857639Z lvl=info msg="Open store (start)" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open op_event=start
influxdb     | ts=2020-06-11T15:31:35.949765Z lvl=info msg="Opened file" log_id=0NK~yfj0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/5/000000003-000000002.tsm id=0 duration=1.935ms
influxdb     | ts=2020-06-11T15:31:35.952848Z lvl=info msg="Opened file" log_id=0NK~yfj0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/9/000000001-000000001.tsm id=0 duration=0.565ms
influxdb     | ts=2020-06-11T15:31:35.951919Z lvl=info msg="Opened file" log_id=0NK~yfj0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/1/000000001-000000001.tsm id=0 duration=0.955ms
nginx        | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
influxdb     | ts=2020-06-11T15:31:35.966456Z lvl=info msg="Reading file" log_id=0NK~yfj0000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/9/_00005.wal size=10494861
influxdb     | ts=2020-06-11T15:31:35.981209Z lvl=info msg="Opened file" log_id=0NK~yfj0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/2/000000003-000000002.tsm id=0 duration=8.400ms
influxdb     | ts=2020-06-11T15:31:36.238930Z lvl=info msg="Opened shard" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/_internal/monitor/2 duration=327.618ms
influxdb     | ts=2020-06-11T15:31:36.238978Z lvl=info msg="Opened shard" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/_internal/monitor/5 duration=327.753ms
influxdb     | ts=2020-06-11T15:31:36.238933Z lvl=info msg="Opened shard" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/_internal/monitor/1 duration=327.667ms
influxdb     | ts=2020-06-11T15:31:36.243050Z lvl=info msg="Opened file" log_id=0NK~yfj0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/6/000000003-000000002.tsm id=0 duration=0.668ms
influxdb     | ts=2020-06-11T15:31:36.246036Z lvl=info msg="Opened file" log_id=0NK~yfj0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/_internal/monitor/7/000000005-000000002.tsm id=0 duration=0.891ms
influxdb     | ts=2020-06-11T15:31:36.290382Z lvl=info msg="Opened file" log_id=0NK~yfj0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/telegraf/autogen/4/000000014-000000002.tsm id=0 duration=38.120ms
influxdb     | ts=2020-06-11T15:31:36.334572Z lvl=info msg="Opened shard" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/_internal/monitor/6 duration=95.392ms
influxdb     | ts=2020-06-11T15:31:36.366455Z lvl=info msg="Opened shard" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/_internal/monitor/7 duration=127.290ms
influxdb     | ts=2020-06-11T15:31:36.372729Z lvl=info msg="Opened file" log_id=0NK~yfj0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/telegraf/autogen/3/000000001-000000001.tsm id=0 duration=5.548ms
influxdb     | ts=2020-06-11T15:31:36.373279Z lvl=info msg="Opened file" log_id=0NK~yfj0000 engine=tsm1 service=filestore path=/var/lib/influxdb/data/telegraf/autogen/8/000000010-000000002.tsm id=0 duration=1.028ms
influxdb     | ts=2020-06-11T15:31:36.522770Z lvl=info msg="Opened shard" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/telegraf/autogen/8 duration=156.173ms
influxdb     | ts=2020-06-11T15:31:36.723230Z lvl=info msg="Opened shard" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/telegraf/autogen/3 duration=388.458ms
influxdb     | ts=2020-06-11T15:31:37.027732Z lvl=info msg="Opened shard" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/telegraf/autogen/4 duration=787.474ms
telegraf     | 2020-06-11T15:31:38Z I! Starting Telegraf 1.14.4
telegraf     | 2020-06-11T15:31:38Z I! Using config file: /etc/telegraf/telegraf.conf
telegraf     | 2020-06-11T15:31:38Z I! [inputs.mqtt_consumer] Connected [tcp://eu.thethings.network:1883]
telegraf     | 2020-06-11T15:31:38Z I! [inputs.mqtt_consumer] Connected [tcp://eu.thethings.network:1883]
telegraf     | 2020-06-11T15:31:38Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
telegraf     | > system,host=5d55d71ced7c load1=0.59,load15=0.13,load5=0.22,n_cpus=4i,n_users=0i 1591889499000000000
telegraf     | > system,host=5d55d71ced7c uptime=272876i 1591889499000000000
node-red     |
node-red     | > node-red-docker@1.0.6 start /usr/src/node-red
node-red     | > node $NODE_OPTIONS node_modules/node-red/red.js $FLOWS "--userDir" "/data"
node-red     |
node-red     | 11 Jun 16:31:38 - [info]
node-red     |
node-red     | Welcome to Node-RED
node-red     | ===================
node-red     |
node-red     | 11 Jun 16:31:38 - [info] Node-RED version: v1.0.6
node-red     | 11 Jun 16:31:38 - [info] Node.js  version: v10.20.1
node-red     | 11 Jun 16:31:38 - [info] Linux 4.19.118-v7l+ arm LE
node-red     | 11 Jun 16:31:39 - [info] Loading palette nodes
nginx        | 10-listen-on-ipv6-by-default.sh: IPv6 listen already enabled, exiting
nginx        | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
telegraf     | > system,host=5d55d71ced7c uptime_format="3 days,  3:47" 1591889499000000000
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889498820552388
telegraf     | > cpu,cpu=cpu0,host=5d55d71ced7c usage_guest=0,usage_guest_nice=0,usage_idle=6.122448984682886,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=6.122448979021745,usage_user=87.75510203211911 1591889499000000000
telegraf     | > cpu,cpu=cpu1,host=5d55d71ced7c usage_guest=0,usage_guest_nice=0,usage_idle=2.0000000016298145,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=35.99999999582178,usage_user=61.999999992862286 1591889499000000000
telegraf     | > cpu,cpu=cpu2,host=5d55d71ced7c usage_guest=0,usage_guest_nice=0,usage_idle=2.0408163282276286,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=26.530612242272888,usage_user=71.4285714215182 1591889499000000000
telegraf     | > cpu,cpu=cpu3,host=5d55d71ced7c usage_guest=0,usage_guest_nice=0,usage_idle=12.244897958941243,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=16.32653061254036,usage_user=71.42857142981768 1591889499000000000
telegraf     | > cpu,cpu=cpu-total,host=5d55d71ced7c usage_guest=0,usage_guest_nice=0,usage_idle=5.583756338638321,usage_iowait=0,usage_irq=0,usage_nice=0,usage_softirq=0,usage_steal=0,usage_system=21.319796954620784,usage_user=73.09644670147783 1591889499000000000
telegraf     | > disk,device=sda2,fstype=fuseblk,host=5d55d71ced7c,mode=rw,path=/etc/telegraf free=119548018688i,inodes_free=912079i,inodes_total=914136i,inodes_used=2057i,total=119822876672i,used=274857984i,used_percent=0.2293869014281714 1591889499000000000
telegraf     | > disk,device=mmcblk0p2,fstype=ext4,host=5d55d71ced7c,mode=rw,path=/etc/resolv.conf free=25558777856i,inodes_free=1752955i,inodes_total=1899328i,inodes_used=146373i,total=31120502784i,used=4264648704i,used_percent=14.299660353984489 1591889499000000000
telegraf     | > disk,device=mmcblk0p2,fstype=ext4,host=5d55d71ced7c,mode=rw,path=/etc/hostname free=25558777856i,inodes_free=1752955i,inodes_total=1899328i,inodes_used=146373i,total=31120502784i,used=4264648704i,used_percent=14.299660353984489 1591889499000000000
telegraf     | > disk,device=mmcblk0p2,fstype=ext4,host=5d55d71ced7c,mode=rw,path=/etc/hosts free=25558777856i,inodes_free=1752955i,inodes_total=1899328i,inodes_used=146373i,total=31120502784i,used=4264648704i,used_percent=14.299660353984489 1591889499000000000
telegraf     | > kernel,host=5d55d71ced7c boot_time=1591616622i,context_switches=170923611i,entropy_avail=3597i,interrupts=96024889i,processes_forked=568205i 1591889499000000000
telegraf     | > mem,host=5d55d71ced7c active=943988736i,available=3495632896i,available_percent=85.34806227598567,buffered=241991680i,cached=1904562176i,commit_limit=2152722432i,committed_as=2057998336i,dirty=385024i,free=1522171904i,high_free=1235828736i,high_total=3342860288i,huge_page_size=0i,huge_pages_free=0i,huge_pages_total=0i,inactive=1393422336i,low_free=286343168i,low_total=752877568i,mapped=223375360i,page_tables=5177344i,shared=51154944i,slab=169033728i,sreclaimable=112640000i,sunreclaim=56393728i,swap_cached=0i,swap_free=104853504i,swap_total=104853504i,total=4095737856i,used=427012096i,used_percent=10.425767249103943,vmalloc_chunk=0i,vmalloc_total=251658240i,vmalloc_used=0i,wired=0i,write_back=0i,write_back_tmp=0i 1591889499000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=803,condition_main=Clouds,country=GB,forecast=3h,host=5d55d71ced7c cloudiness=69i,condition_description="broken clouds",condition_icon="04d",humidity=64i,pressure=1024,rain=0,temperature=13.84,wind_degrees=52,wind_speed=5.59 1591898400000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=6h,host=5d55d71ced7c cloudiness=92i,condition_description="overcast clouds",condition_icon="04n",humidity=71i,pressure=1022,rain=0,temperature=12.92,wind_degrees=52,wind_speed=6.12 1591909200000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=9h,host=5d55d71ced7c cloudiness=97i,condition_description="overcast clouds",condition_icon="04n",humidity=73i,pressure=1021,rain=0,temperature=12.19,wind_degrees=43,wind_speed=6.73 1591920000000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=12h,host=5d55d71ced7c cloudiness=100i,condition_description="overcast clouds",condition_icon="04n",humidity=77i,pressure=1018,rain=0,temperature=11.54,wind_degrees=49,wind_speed=6.45 1591930800000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=15h,host=5d55d71ced7c cloudiness=100i,condition_description="overcast clouds",condition_icon="04d",humidity=78i,pressure=1017,rain=0,temperature=11.73,wind_degrees=50,wind_speed=6.71 1591941600000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=18h,host=5d55d71ced7c cloudiness=100i,condition_description="overcast clouds",condition_icon="04d",humidity=78i,pressure=1016,rain=0,temperature=12.29,wind_degrees=50,wind_speed=6.63 1591952400000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=21h,host=5d55d71ced7c cloudiness=100i,condition_description="overcast clouds",condition_icon="04d",humidity=78i,pressure=1015,rain=0,temperature=12.66,wind_degrees=56,wind_speed=7.01 1591963200000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=24h,host=5d55d71ced7c cloudiness=99i,condition_description="overcast clouds",condition_icon="04d",humidity=82i,pressure=1014,rain=0,temperature=12.28,wind_degrees=57,wind_speed=6.74 1591974000000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=27h,host=5d55d71ced7c cloudiness=100i,condition_description="light rain",condition_icon="10d",humidity=92i,pressure=1013,rain=0.55,temperature=10.85,wind_degrees=51,wind_speed=6.07 1591984800000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=30h,host=5d55d71ced7c cloudiness=100i,condition_description="light rain",condition_icon="10n",humidity=94i,pressure=1013,rain=0.38,temperature=10.95,wind_degrees=59,wind_speed=4.74 1591995600000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=33h,host=5d55d71ced7c cloudiness=100i,condition_description="overcast clouds",condition_icon="04n",humidity=95i,pressure=1013,rain=0,temperature=11.18,wind_degrees=58,wind_speed=4.56 1592006400000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=36h,host=5d55d71ced7c cloudiness=100i,condition_description="light rain",condition_icon="10n",humidity=95i,pressure=1012,rain=0.16,temperature=11.26,wind_degrees=41,wind_speed=3.48 1592017200000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=39h,host=5d55d71ced7c cloudiness=100i,condition_description="light rain",condition_icon="10d",humidity=96i,pressure=1012,rain=0.33,temperature=11.9,wind_degrees=47,wind_speed=3.49 1592028000000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=42h,host=5d55d71ced7c cloudiness=100i,condition_description="light rain",condition_icon="10d",humidity=94i,pressure=1013,rain=0.11,temperature=13.26,wind_degrees=54,wind_speed=4.65 1592038800000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=45h,host=5d55d71ced7c cloudiness=100i,condition_description="light rain",condition_icon="10d",humidity=96i,pressure=1013,rain=0.97,temperature=13.23,wind_degrees=55,wind_speed=4.96 1592049600000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=48h,host=5d55d71ced7c cloudiness=100i,condition_description="light rain",condition_icon="10d",humidity=95i,pressure=1013,rain=0.59,temperature=13.52,wind_degrees=61,wind_speed=5.13 1592060400000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=51h,host=5d55d71ced7c cloudiness=100i,condition_description="light rain",condition_icon="10d",humidity=94i,pressure=1013,rain=0.1,temperature=13.31,wind_degrees=59,wind_speed=4.76 1592071200000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=54h,host=5d55d71ced7c cloudiness=95i,condition_description="overcast clouds",condition_icon="04n",humidity=96i,pressure=1015,rain=0,temperature=12.88,wind_degrees=61,wind_speed=3.82 1592082000000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=57h,host=5d55d71ced7c cloudiness=97i,condition_description="overcast clouds",condition_icon="04n",humidity=96i,pressure=1015,rain=0,temperature=12.66,wind_degrees=58,wind_speed=3.09 1592092800000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=60h,host=5d55d71ced7c cloudiness=95i,condition_description="overcast clouds",condition_icon="04n",humidity=97i,pressure=1015,rain=0,temperature=12.56,wind_degrees=51,wind_speed=2.29 1592103600000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=63h,host=5d55d71ced7c cloudiness=97i,condition_description="overcast clouds",condition_icon="04d",humidity=96i,pressure=1015,rain=0,temperature=13.03,wind_degrees=56,wind_speed=2.13 1592114400000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=66h,host=5d55d71ced7c cloudiness=100i,condition_description="overcast clouds",condition_icon="04d",humidity=93i,pressure=1016,rain=0,temperature=14.03,wind_degrees=66,wind_speed=2.37 1592125200000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=69h,host=5d55d71ced7c cloudiness=99i,condition_description="overcast clouds",condition_icon="04d",humidity=83i,pressure=1015,rain=0,temperature=16.24,wind_degrees=64,wind_speed=3.29 1592136000000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=803,condition_main=Clouds,country=GB,forecast=72h,host=5d55d71ced7c cloudiness=60i,condition_description="broken clouds",condition_icon="04d",humidity=83i,pressure=1014,rain=0,temperature=16.7,wind_degrees=67,wind_speed=4.5 1592146800000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=75h,host=5d55d71ced7c cloudiness=67i,condition_description="light rain",condition_icon="10d",humidity=91i,pressure=1015,rain=0.7,temperature=15.14,wind_degrees=62,wind_speed=3.69 1592157600000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=78h,host=5d55d71ced7c cloudiness=92i,condition_description="light rain",condition_icon="10n",humidity=94i,pressure=1016,rain=0.12,temperature=13.48,wind_degrees=53,wind_speed=2.77 1592168400000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=81h,host=5d55d71ced7c cloudiness=95i,condition_description="overcast clouds",condition_icon="04n",humidity=94i,pressure=1015,rain=0,temperature=12.87,wind_degrees=50,wind_speed=2.27 1592179200000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=84h,host=5d55d71ced7c cloudiness=89i,condition_description="overcast clouds",condition_icon="04n",humidity=95i,pressure=1016,rain=0,temperature=12.49,wind_degrees=59,wind_speed=1.64 1592190000000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=87h,host=5d55d71ced7c cloudiness=91i,condition_description="overcast clouds",condition_icon="04d",humidity=92i,pressure=1015,rain=0,temperature=13.4,wind_degrees=98,wind_speed=1.31 1592200800000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=803,condition_main=Clouds,country=GB,forecast=90h,host=5d55d71ced7c cloudiness=84i,condition_description="broken clouds",condition_icon="04d",humidity=80i,pressure=1015,rain=0,temperature=15.99,wind_degrees=47,wind_speed=1.27 1592211600000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=93h,host=5d55d71ced7c cloudiness=81i,condition_description="light rain",condition_icon="10d",humidity=75i,pressure=1015,rain=0.33,temperature=17.19,wind_degrees=71,wind_speed=2.58 1592222400000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=96h,host=5d55d71ced7c cloudiness=49i,condition_description="light rain",condition_icon="10d",humidity=75i,pressure=1014,rain=0.3,temperature=17.49,wind_degrees=59,wind_speed=2.95 1592233200000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=99h,host=5d55d71ced7c cloudiness=43i,condition_description="light rain",condition_icon="10d",humidity=79i,pressure=1013,rain=1.13,temperature=17.13,wind_degrees=9,wind_speed=0.38 1592244000000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=802,condition_main=Clouds,country=GB,forecast=102h,host=5d55d71ced7c cloudiness=27i,condition_description="scattered clouds",condition_icon="03d",humidity=84i,pressure=1014,rain=0,temperature=14.15,wind_degrees=199,wind_speed=0.45 1592254800000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=802,condition_main=Clouds,country=GB,forecast=105h,host=5d55d71ced7c cloudiness=33i,condition_description="scattered clouds",condition_icon="03n",humidity=83i,pressure=1013,rain=0,temperature=12.65,wind_degrees=226,wind_speed=1.19 1592265600000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=804,condition_main=Clouds,country=GB,forecast=108h,host=5d55d71ced7c cloudiness=92i,condition_description="overcast clouds",condition_icon="04n",humidity=84i,pressure=1013,rain=0,temperature=11.98,wind_degrees=233,wind_speed=1.44 1592276400000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=803,condition_main=Clouds,country=GB,forecast=111h,host=5d55d71ced7c cloudiness=71i,condition_description="broken clouds",condition_icon="04d",humidity=81i,pressure=1013,rain=0,temperature=13.62,wind_degrees=214,wind_speed=0.69 1592287200000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=803,condition_main=Clouds,country=GB,forecast=114h,host=5d55d71ced7c cloudiness=55i,condition_description="broken clouds",condition_icon="04d",humidity=68i,pressure=1012,rain=0,temperature=16.82,wind_degrees=211,wind_speed=0.43 1592298000000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=500,condition_main=Rain,country=GB,forecast=117h,host=5d55d71ced7c cloudiness=67i,condition_description="light rain",condition_icon="10d",humidity=69i,pressure=1011,rain=2.1,temperature=17.78,wind_degrees=107,wind_speed=2.48 1592308800000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=501,condition_main=Rain,country=GB,forecast=120h,host=5d55d71ced7c cloudiness=99i,condition_description="moderate rain",condition_icon="10d",humidity=66i,pressure=1010,rain=4.43,temperature=18.54,wind_degrees=203,wind_speed=1.09 1592319600000000000
telegraf     | > edinburgh_weather,city=Edinburgh,city_id=2650225,condition_id=803,condition_main=Clouds,country=GB,forecast=*,host=5d55d71ced7c cloudiness=63i,condition_description="broken clouds",condition_icon="04d",humidity=58i,pressure=1024,rain=0,sunrise=1591846048000000000i,sunset=1591909066000000000i,temperature=15.52,visibility=10000i,wind_degrees=70,wind_speed=8.2 1591888175000000000
telegraf     | > diskio,host=5d55d71ced7c,name=mmcblk0p1 io_time=1220i,iops_in_progress=0i,merged_reads=3879i,merged_writes=0i,read_bytes=37559296i,read_time=1450i,reads=573i,weighted_io_time=1400i,write_bytes=1536i,write_time=11i,writes=3i 1591889500000000000
telegraf     | > diskio,host=5d55d71ced7c,name=mmcblk0p2 io_time=603190i,iops_in_progress=0i,merged_reads=7218i,merged_writes=445362i,read_bytes=607167488i,read_time=53684i,reads=15804i,weighted_io_time=961460i,write_bytes=4798275584i,write_time=8279100i,writes=251982i 1591889500000000000
telegraf     | > diskio,host=5d55d71ced7c,name=sda io_time=22600i,iops_in_progress=0i,merged_reads=0i,merged_writes=266701i,read_bytes=140591104i,read_time=8265i,reads=12429i,weighted_io_time=57140i,write_bytes=1447092224i,write_time=300673i,writes=119888i 1591889500000000000
telegraf     | > diskio,host=5d55d71ced7c,name=sda1 io_time=180i,iops_in_progress=0i,merged_reads=0i,merged_writes=0i,read_bytes=5316608i,read_time=275i,reads=146i,weighted_io_time=300i,write_bytes=0i,write_time=0i,writes=0i 1591889500000000000
telegraf     | > diskio,host=5d55d71ced7c,name=sda2 io_time=22440i,iops_in_progress=0i,merged_reads=0i,merged_writes=266701i,read_bytes=134529024i,read_time=7935i,reads=12223i,weighted_io_time=56810i,write_bytes=1447092224i,write_time=60562i,writes=86593i 1591889500000000000
telegraf     | > diskio,host=5d55d71ced7c,name=mmcblk0 io_time=603420i,iops_in_progress=0i,merged_reads=11097i,merged_writes=445362i,read_bytes=645267456i,read_time=55190i,reads=16397i,weighted_io_time=961470i,write_bytes=4798277120i,write_time=8279112i,writes=251985i 1591889500000000000
telegraf     | > processes,host=5d55d71ced7c blocked=0i,dead=0i,idle=0i,paging=0i,running=0i,sleeping=1i,stopped=0i,total=1i,total_threads=13i,unknown=0i,zombies=0i 1591889500000000000
telegraf     | > swap,host=5d55d71ced7c free=104853504i,total=104853504i,used=0i,used_percent=0 1591889500000000000
telegraf     | > swap,host=5d55d71ced7c in=0i,out=0i 1591889500000000000
telegraf     | > docker,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 n_containers=8i,n_containers_paused=0i,n_containers_running=5i,n_containers_stopped=3i,n_cpus=4i,n_goroutines=126i,n_images=18i,n_listener_events=0i,n_used_file_descriptors=93i 1591889500000000000
telegraf     | > docker,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 memory_total=4095737856i 1591889500000000000
nginx        | /docker-entrypoint.sh: Configuration complete; ready for start up
grafana      | t=2020-06-11T16:31:40+0100 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket=
telegraf     | > docker_container_status,com.docker.compose.config-hash=d3d7144a9de4079e69fbc33c89326b91f5df423cc2b2302ac8a908d8986bb5f2,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=influxdb,com.docker.compose.version=1.26.0,container_image=influxdb,container_name=influxdb,container_status=running,container_version=unknown,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="3f421d63b2e5c4302d7806cad3f309fa39b67940671d78f009c276446645d842",exitcode=0i,finished_at=1591889444252219479i,oomkilled=false,pid=20890i,started_at=1591889495648587164i,uptime_ns=5478566530i 1591889501000000000
telegraf     | > docker_container_mem,com.docker.compose.config-hash=d3d7144a9de4079e69fbc33c89326b91f5df423cc2b2302ac8a908d8986bb5f2,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=influxdb,com.docker.compose.version=1.26.0,container_image=influxdb,container_name=influxdb,container_status=running,container_version=unknown,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 active_anon=49655808i,active_file=0i,cache=0i,container_id="3f421d63b2e5c4302d7806cad3f309fa39b67940671d78f009c276446645d842",hierarchical_memory_limit=8796093018112i,inactive_anon=0i,inactive_file=3866624i,limit=4095737856i,mapped_file=0i,max_usage=54800384i,pgfault=19404i,pgmajfault=0i,pgpgin=14784i,pgpgout=1763i,rss=53448704i,rss_huge=0i,total_active_anon=49655808i,total_active_file=0i,total_cache=0i,total_inactive_anon=0i,total_inactive_file=3866624i,total_mapped_file=0i,total_pgfault=19404i,total_pgmajfault=0i,total_pgpgin=14784i,total_pgpgout=1763i,total_rss=53448704i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=54800384i,usage_percent=1.3379856310803893,writeback=0i 1591889501000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=d3d7144a9de4079e69fbc33c89326b91f5df423cc2b2302ac8a908d8986bb5f2,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=influxdb,com.docker.compose.version=1.26.0,container_image=influxdb,container_name=influxdb,container_status=running,container_version=unknown,cpu=cpu-total,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="3f421d63b2e5c4302d7806cad3f309fa39b67940671d78f009c276446645d842",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=370000000i,usage_in_usermode=5910000000i,usage_percent=120.77150133333335,usage_system=1089127700000000i,usage_total=6180473927i 1591889501000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=d3d7144a9de4079e69fbc33c89326b91f5df423cc2b2302ac8a908d8986bb5f2,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=influxdb,com.docker.compose.version=1.26.0,container_image=influxdb,container_name=influxdb,container_status=running,container_version=unknown,cpu=cpu0,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="3f421d63b2e5c4302d7806cad3f309fa39b67940671d78f009c276446645d842",usage_total=1881621230i 1591889501000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=d3d7144a9de4079e69fbc33c89326b91f5df423cc2b2302ac8a908d8986bb5f2,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=influxdb,com.docker.compose.version=1.26.0,container_image=influxdb,container_name=influxdb,container_status=running,container_version=unknown,cpu=cpu1,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="3f421d63b2e5c4302d7806cad3f309fa39b67940671d78f009c276446645d842",usage_total=1172798754i 1591889501000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=d3d7144a9de4079e69fbc33c89326b91f5df423cc2b2302ac8a908d8986bb5f2,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=influxdb,com.docker.compose.version=1.26.0,container_image=influxdb,container_name=influxdb,container_status=running,container_version=unknown,cpu=cpu2,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="3f421d63b2e5c4302d7806cad3f309fa39b67940671d78f009c276446645d842",usage_total=2278904152i 1591889501000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=d3d7144a9de4079e69fbc33c89326b91f5df423cc2b2302ac8a908d8986bb5f2,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=influxdb,com.docker.compose.version=1.26.0,container_image=influxdb,container_name=influxdb,container_status=running,container_version=unknown,cpu=cpu3,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="3f421d63b2e5c4302d7806cad3f309fa39b67940671d78f009c276446645d842",usage_total=847149791i 1591889501000000000
telegraf     | > docker_container_net,com.docker.compose.config-hash=d3d7144a9de4079e69fbc33c89326b91f5df423cc2b2302ac8a908d8986bb5f2,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=influxdb,com.docker.compose.version=1.26.0,container_image=influxdb,container_name=influxdb,container_status=running,container_version=unknown,engine_host=grafana,host=5d55d71ced7c,network=eth0,server_version=19.03.11 container_id="3f421d63b2e5c4302d7806cad3f309fa39b67940671d78f009c276446645d842",rx_bytes=8336i,rx_dropped=0i,rx_errors=0i,rx_packets=51i,tx_bytes=687i,tx_dropped=0i,tx_errors=0i,tx_packets=4i 1591889501000000000
influxdb     | ts=2020-06-11T15:31:41.294483Z lvl=info msg="Reading file" log_id=0NK~yfj0000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/9/_00006.wal size=10493311
node-red     | 11 Jun 16:31:41 - [info] Settings file  : /data/settings.js
node-red     | 11 Jun 16:31:41 - [info] Context store  : 'default' [module=memory]
node-red     | 11 Jun 16:31:41 - [info] User directory : /data
node-red     | 11 Jun 16:31:41 - [warn] Projects disabled : editorTheme.projects.enabled=false
node-red     | 11 Jun 16:31:41 - [info] Flows file     : /data/flows.json
node-red     | 11 Jun 16:31:41 - [info] Creating new flow file
node-red     | 11 Jun 16:31:41 - [info] Starting flows
node-red     | 11 Jun 16:31:41 - [info] Started flows
node-red     | 11 Jun 16:31:42 - [info] Server now running at http://127.0.0.1:1880/node-red/
telegraf     | > docker_container_status,com.docker.compose.config-hash=b0fa3fc43ba94087e2331a47c3554d1e5c7ebd8204e1ebbe4131c71c47ddc1b4,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=grafana,com.docker.compose.version=1.26.0,container_image=grafana/grafana,container_name=grafana,container_status=running,container_version=unknown,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="29c377392cf5c89483202d96794f8633f99b23c4b144e9a68d510126505d8d51",exitcode=0i,finished_at=1591889442299027323i,oomkilled=false,pid=21249i,started_at=1591889498083788174i,uptime_ns=4065193678i 1591889502000000000
telegraf     | > docker_container_mem,com.docker.compose.config-hash=b0fa3fc43ba94087e2331a47c3554d1e5c7ebd8204e1ebbe4131c71c47ddc1b4,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=grafana,com.docker.compose.version=1.26.0,container_image=grafana/grafana,container_name=grafana,container_status=running,container_version=unknown,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 active_anon=11448320i,active_file=0i,cache=0i,container_id="29c377392cf5c89483202d96794f8633f99b23c4b144e9a68d510126505d8d51",hierarchical_memory_limit=8796093018112i,inactive_anon=0i,inactive_file=1155072i,limit=4095737856i,mapped_file=0i,max_usage=14548992i,pgfault=7194i,pgmajfault=0i,pgpgin=4818i,pgpgout=1729i,rss=12279808i,rss_huge=0i,total_active_anon=11448320i,total_active_file=0i,total_cache=0i,total_inactive_anon=0i,total_inactive_file=1155072i,total_mapped_file=0i,total_pgfault=7194i,total_pgmajfault=0i,total_pgpgin=4818i,total_pgpgout=1729i,total_rss=12279808i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=14430208i,usage_percent=0.35232254864311313,writeback=0i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=b0fa3fc43ba94087e2331a47c3554d1e5c7ebd8204e1ebbe4131c71c47ddc1b4,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=grafana,com.docker.compose.version=1.26.0,container_image=grafana/grafana,container_name=grafana,container_status=running,container_version=unknown,cpu=cpu-total,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="29c377392cf5c89483202d96794f8633f99b23c4b144e9a68d510126505d8d51",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=350000000i,usage_in_usermode=1770000000i,usage_percent=0.14964267326732672,usage_system=1089131810000000i,usage_total=2113147302i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=b0fa3fc43ba94087e2331a47c3554d1e5c7ebd8204e1ebbe4131c71c47ddc1b4,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=grafana,com.docker.compose.version=1.26.0,container_image=grafana/grafana,container_name=grafana,container_status=running,container_version=unknown,cpu=cpu0,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="29c377392cf5c89483202d96794f8633f99b23c4b144e9a68d510126505d8d51",usage_total=849226421i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=b0fa3fc43ba94087e2331a47c3554d1e5c7ebd8204e1ebbe4131c71c47ddc1b4,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=grafana,com.docker.compose.version=1.26.0,container_image=grafana/grafana,container_name=grafana,container_status=running,container_version=unknown,cpu=cpu1,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="29c377392cf5c89483202d96794f8633f99b23c4b144e9a68d510126505d8d51",usage_total=534698546i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=b0fa3fc43ba94087e2331a47c3554d1e5c7ebd8204e1ebbe4131c71c47ddc1b4,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=grafana,com.docker.compose.version=1.26.0,container_image=grafana/grafana,container_name=grafana,container_status=running,container_version=unknown,cpu=cpu2,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="29c377392cf5c89483202d96794f8633f99b23c4b144e9a68d510126505d8d51",usage_total=228634183i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=b0fa3fc43ba94087e2331a47c3554d1e5c7ebd8204e1ebbe4131c71c47ddc1b4,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=grafana,com.docker.compose.version=1.26.0,container_image=grafana/grafana,container_name=grafana,container_status=running,container_version=unknown,cpu=cpu3,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="29c377392cf5c89483202d96794f8633f99b23c4b144e9a68d510126505d8d51",usage_total=500588152i 1591889502000000000
telegraf     | > docker_container_net,com.docker.compose.config-hash=b0fa3fc43ba94087e2331a47c3554d1e5c7ebd8204e1ebbe4131c71c47ddc1b4,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=grafana,com.docker.compose.version=1.26.0,container_image=grafana/grafana,container_name=grafana,container_status=running,container_version=unknown,engine_host=grafana,host=5d55d71ced7c,network=eth0,server_version=19.03.11 container_id="29c377392cf5c89483202d96794f8633f99b23c4b144e9a68d510126505d8d51",rx_bytes=15207i,rx_dropped=0i,rx_errors=0i,rx_packets=53i,tx_bytes=4196i,tx_dropped=0i,tx_errors=0i,tx_packets=38i 1591889502000000000
telegraf     | > docker_container_status,authors=Dave\ Conway-Jones\,\ Nick\ O'Leary\,\ James\ Thomas\,\ Raymond\ Mouthaan,com.docker.compose.config-hash=dfae2d549c9ac5df2848f6e037f25815ae2cf70ec98e4e9c3953c39b046ffbdd,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=node-red,com.docker.compose.version=1.26.0,container_image=nodered/node-red,container_name=node-red,container_status=running,container_version=latest,engine_host=grafana,host=5d55d71ced7c,org.label-schema.arch=arm32v7,org.label-schema.build-date=2020-05-21T10:09:24Z,org.label-schema.description=Low-code\ programming\ for\ event-driven\ applications.,org.label-schema.docker.dockerfile=.docker/Dockerfile.alpine,org.label-schema.license=Apache-2.0,org.label-schema.name=Node-RED,org.label-schema.url=https://nodered.org,org.label-schema.vcs-ref=2d97c989f158d0710c723c8b5b00cb5760fd221d,org.label-schema.vcs-type=Git,org.label-schema.vcs-url=https://github.com/node-red/node-red-docker,org.label-schema.version=1.0.6-2,server_version=19.03.11 container_id="1c8d6c4a9c5e57de47e843b6004a61a502a643d3f578dc3aa7d50556ecc87bcb",exitcode=0i,finished_at=1591889442286792728i,oomkilled=false,pid=20808i,started_at=1591889495451439905i,uptime_ns=6705239720i 1591889502000000000
telegraf     | > docker_container_health,authors=Dave\ Conway-Jones\,\ Nick\ O'Leary\,\ James\ Thomas\,\ Raymond\ Mouthaan,com.docker.compose.config-hash=dfae2d549c9ac5df2848f6e037f25815ae2cf70ec98e4e9c3953c39b046ffbdd,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=node-red,com.docker.compose.version=1.26.0,container_image=nodered/node-red,container_name=node-red,container_status=running,container_version=latest,engine_host=grafana,host=5d55d71ced7c,org.label-schema.arch=arm32v7,org.label-schema.build-date=2020-05-21T10:09:24Z,org.label-schema.description=Low-code\ programming\ for\ event-driven\ applications.,org.label-schema.docker.dockerfile=.docker/Dockerfile.alpine,org.label-schema.license=Apache-2.0,org.label-schema.name=Node-RED,org.label-schema.url=https://nodered.org,org.label-schema.vcs-ref=2d97c989f158d0710c723c8b5b00cb5760fd221d,org.label-schema.vcs-type=Git,org.label-schema.vcs-url=https://github.com/node-red/node-red-docker,org.label-schema.version=1.0.6-2,server_version=19.03.11 failing_streak=0i,health_status="starting" 1591889502000000000
telegraf     | > docker_container_mem,authors=Dave\ Conway-Jones\,\ Nick\ O'Leary\,\ James\ Thomas\,\ Raymond\ Mouthaan,com.docker.compose.config-hash=dfae2d549c9ac5df2848f6e037f25815ae2cf70ec98e4e9c3953c39b046ffbdd,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=node-red,com.docker.compose.version=1.26.0,container_image=nodered/node-red,container_name=node-red,container_status=running,container_version=latest,engine_host=grafana,host=5d55d71ced7c,org.label-schema.arch=arm32v7,org.label-schema.build-date=2020-05-21T10:09:24Z,org.label-schema.description=Low-code\ programming\ for\ event-driven\ applications.,org.label-schema.docker.dockerfile=.docker/Dockerfile.alpine,org.label-schema.license=Apache-2.0,org.label-schema.name=Node-RED,org.label-schema.url=https://nodered.org,org.label-schema.vcs-ref=2d97c989f158d0710c723c8b5b00cb5760fd221d,org.label-schema.vcs-type=Git,org.label-schema.vcs-url=https://github.com/node-red/node-red-docker,org.label-schema.version=1.0.6-2,server_version=19.03.11 active_anon=49729536i,active_file=4096i,cache=0i,container_id="1c8d6c4a9c5e57de47e843b6004a61a502a643d3f578dc3aa7d50556ecc87bcb",hierarchical_memory_limit=8796093018112i,inactive_anon=0i,inactive_file=0i,limit=4095737856i,mapped_file=0i,max_usage=53403648i,pgfault=30426i,pgmajfault=0i,pgpgin=23133i,pgpgout=11051i,rss=49528832i,rss_huge=0i,total_active_anon=49729536i,total_active_file=4096i,total_cache=0i,total_inactive_anon=0i,total_inactive_file=0i,total_mapped_file=0i,total_pgfault=30426i,total_pgmajfault=0i,total_pgpgin=23133i,total_pgpgout=11051i,total_rss=49528832i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=52867072i,usage_percent=1.2907826100870454,writeback=0i 1591889502000000000
telegraf     | > docker_container_cpu,authors=Dave\ Conway-Jones\,\ Nick\ O'Leary\,\ James\ Thomas\,\ Raymond\ Mouthaan,com.docker.compose.config-hash=dfae2d549c9ac5df2848f6e037f25815ae2cf70ec98e4e9c3953c39b046ffbdd,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=node-red,com.docker.compose.version=1.26.0,container_image=nodered/node-red,container_name=node-red,container_status=running,container_version=latest,cpu=cpu-total,engine_host=grafana,host=5d55d71ced7c,org.label-schema.arch=arm32v7,org.label-schema.build-date=2020-05-21T10:09:24Z,org.label-schema.description=Low-code\ programming\ for\ event-driven\ applications.,org.label-schema.docker.dockerfile=.docker/Dockerfile.alpine,org.label-schema.license=Apache-2.0,org.label-schema.name=Node-RED,org.label-schema.url=https://nodered.org,org.label-schema.vcs-ref=2d97c989f158d0710c723c8b5b00cb5760fd221d,org.label-schema.vcs-type=Git,org.label-schema.vcs-url=https://github.com/node-red/node-red-docker,org.label-schema.version=1.0.6-2,server_version=19.03.11 container_id="1c8d6c4a9c5e57de47e843b6004a61a502a643d3f578dc3aa7d50556ecc87bcb",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=1110000000i,usage_in_usermode=4970000000i,usage_percent=106.52135831265508,usage_system=1089131810000000i,usage_total=6210753281i 1591889502000000000
telegraf     | > docker_container_cpu,authors=Dave\ Conway-Jones\,\ Nick\ O'Leary\,\ James\ Thomas\,\ Raymond\ Mouthaan,com.docker.compose.config-hash=dfae2d549c9ac5df2848f6e037f25815ae2cf70ec98e4e9c3953c39b046ffbdd,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=node-red,com.docker.compose.version=1.26.0,container_image=nodered/node-red,container_name=node-red,container_status=running,container_version=latest,cpu=cpu0,engine_host=grafana,host=5d55d71ced7c,org.label-schema.arch=arm32v7,org.label-schema.build-date=2020-05-21T10:09:24Z,org.label-schema.description=Low-code\ programming\ for\ event-driven\ applications.,org.label-schema.docker.dockerfile=.docker/Dockerfile.alpine,org.label-schema.license=Apache-2.0,org.label-schema.name=Node-RED,org.label-schema.url=https://nodered.org,org.label-schema.vcs-ref=2d97c989f158d0710c723c8b5b00cb5760fd221d,org.label-schema.vcs-type=Git,org.label-schema.vcs-url=https://github.com/node-red/node-red-docker,org.label-schema.version=1.0.6-2,server_version=19.03.11 container_id="1c8d6c4a9c5e57de47e843b6004a61a502a643d3f578dc3aa7d50556ecc87bcb",usage_total=1073936131i 1591889502000000000
telegraf     | > docker_container_cpu,authors=Dave\ Conway-Jones\,\ Nick\ O'Leary\,\ James\ Thomas\,\ Raymond\ Mouthaan,com.docker.compose.config-hash=dfae2d549c9ac5df2848f6e037f25815ae2cf70ec98e4e9c3953c39b046ffbdd,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=node-red,com.docker.compose.version=1.26.0,container_image=nodered/node-red,container_name=node-red,container_status=running,container_version=latest,cpu=cpu1,engine_host=grafana,host=5d55d71ced7c,org.label-schema.arch=arm32v7,org.label-schema.build-date=2020-05-21T10:09:24Z,org.label-schema.description=Low-code\ programming\ for\ event-driven\ applications.,org.label-schema.docker.dockerfile=.docker/Dockerfile.alpine,org.label-schema.license=Apache-2.0,org.label-schema.name=Node-RED,org.label-schema.url=https://nodered.org,org.label-schema.vcs-ref=2d97c989f158d0710c723c8b5b00cb5760fd221d,org.label-schema.vcs-type=Git,org.label-schema.vcs-url=https://github.com/node-red/node-red-docker,org.label-schema.version=1.0.6-2,server_version=19.03.11 container_id="1c8d6c4a9c5e57de47e843b6004a61a502a643d3f578dc3aa7d50556ecc87bcb",usage_total=2352168450i 1591889502000000000
telegraf     | > docker_container_cpu,authors=Dave\ Conway-Jones\,\ Nick\ O'Leary\,\ James\ Thomas\,\ Raymond\ Mouthaan,com.docker.compose.config-hash=dfae2d549c9ac5df2848f6e037f25815ae2cf70ec98e4e9c3953c39b046ffbdd,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=node-red,com.docker.compose.version=1.26.0,container_image=nodered/node-red,container_name=node-red,container_status=running,container_version=latest,cpu=cpu2,engine_host=grafana,host=5d55d71ced7c,org.label-schema.arch=arm32v7,org.label-schema.build-date=2020-05-21T10:09:24Z,org.label-schema.description=Low-code\ programming\ for\ event-driven\ applications.,org.label-schema.docker.dockerfile=.docker/Dockerfile.alpine,org.label-schema.license=Apache-2.0,org.label-schema.name=Node-RED,org.label-schema.url=https://nodered.org,org.label-schema.vcs-ref=2d97c989f158d0710c723c8b5b00cb5760fd221d,org.label-schema.vcs-type=Git,org.label-schema.vcs-url=https://github.com/node-red/node-red-docker,org.label-schema.version=1.0.6-2,server_version=19.03.11 container_id="1c8d6c4a9c5e57de47e843b6004a61a502a643d3f578dc3aa7d50556ecc87bcb",usage_total=1254329291i 1591889502000000000
telegraf     | > docker_container_cpu,authors=Dave\ Conway-Jones\,\ Nick\ O'Leary\,\ James\ Thomas\,\ Raymond\ Mouthaan,com.docker.compose.config-hash=dfae2d549c9ac5df2848f6e037f25815ae2cf70ec98e4e9c3953c39b046ffbdd,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=node-red,com.docker.compose.version=1.26.0,container_image=nodered/node-red,container_name=node-red,container_status=running,container_version=latest,cpu=cpu3,engine_host=grafana,host=5d55d71ced7c,org.label-schema.arch=arm32v7,org.label-schema.build-date=2020-05-21T10:09:24Z,org.label-schema.description=Low-code\ programming\ for\ event-driven\ applications.,org.label-schema.docker.dockerfile=.docker/Dockerfile.alpine,org.label-schema.license=Apache-2.0,org.label-schema.name=Node-RED,org.label-schema.url=https://nodered.org,org.label-schema.vcs-ref=2d97c989f158d0710c723c8b5b00cb5760fd221d,org.label-schema.vcs-type=Git,org.label-schema.vcs-url=https://github.com/node-red/node-red-docker,org.label-schema.version=1.0.6-2,server_version=19.03.11 container_id="1c8d6c4a9c5e57de47e843b6004a61a502a643d3f578dc3aa7d50556ecc87bcb",usage_total=1530319409i 1591889502000000000
telegraf     | > docker_container_net,authors=Dave\ Conway-Jones\,\ Nick\ O'Leary\,\ James\ Thomas\,\ Raymond\ Mouthaan,com.docker.compose.config-hash=dfae2d549c9ac5df2848f6e037f25815ae2cf70ec98e4e9c3953c39b046ffbdd,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=node-red,com.docker.compose.version=1.26.0,container_image=nodered/node-red,container_name=node-red,container_status=running,container_version=latest,engine_host=grafana,host=5d55d71ced7c,network=eth0,org.label-schema.arch=arm32v7,org.label-schema.build-date=2020-05-21T10:09:24Z,org.label-schema.description=Low-code\ programming\ for\ event-driven\ applications.,org.label-schema.docker.dockerfile=.docker/Dockerfile.alpine,org.label-schema.license=Apache-2.0,org.label-schema.name=Node-RED,org.label-schema.url=https://nodered.org,org.label-schema.vcs-ref=2d97c989f158d0710c723c8b5b00cb5760fd221d,org.label-schema.vcs-type=Git,org.label-schema.vcs-url=https://github.com/node-red/node-red-docker,org.label-schema.version=1.0.6-2,server_version=19.03.11 container_id="1c8d6c4a9c5e57de47e843b6004a61a502a643d3f578dc3aa7d50556ecc87bcb",rx_bytes=7188i,rx_dropped=0i,rx_errors=0i,rx_packets=48i,tx_bytes=687i,tx_dropped=0i,tx_errors=0i,tx_packets=4i 1591889502000000000
telegraf     | > docker_container_status,com.docker.compose.config-hash=0387b682ef63fa7c3bf5d9839c0111704e4af47c3aa223da338a043c5b5e5750,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=mosquitto,com.docker.compose.version=1.26.0,container_image=eclipse-mosquitto,container_name=mosquitto,container_status=running,container_version=unknown,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=grafana,host=5d55d71ced7c,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=19.03.11 container_id="4cc84e7cf5961852326f7aa187ea1e2f7d08c901e445ca7affa4977cc9cb1732",exitcode=0i,finished_at=1591889442216947407i,oomkilled=false,pid=20784i,started_at=1591889495290033944i,uptime_ns=6882245020i 1591889502000000000
telegraf     | > docker_container_mem,com.docker.compose.config-hash=0387b682ef63fa7c3bf5d9839c0111704e4af47c3aa223da338a043c5b5e5750,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=mosquitto,com.docker.compose.version=1.26.0,container_image=eclipse-mosquitto,container_name=mosquitto,container_status=running,container_version=unknown,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=grafana,host=5d55d71ced7c,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=19.03.11 active_anon=315392i,active_file=0i,cache=0i,container_id="4cc84e7cf5961852326f7aa187ea1e2f7d08c901e445ca7affa4977cc9cb1732",hierarchical_memory_limit=8796093018112i,inactive_anon=0i,inactive_file=0i,limit=4095737856i,mapped_file=0i,max_usage=4841472i,pgfault=2079i,pgmajfault=0i,pgpgin=1452i,pgpgout=1387i,rss=417792i,rss_huge=0i,total_active_anon=315392i,total_active_file=0i,total_cache=0i,total_inactive_anon=0i,total_inactive_file=0i,total_mapped_file=0i,total_pgfault=2079i,total_pgmajfault=0i,total_pgpgin=1452i,total_pgpgout=1387i,total_rss=417792i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=880640i,usage_percent=0.021501376088069638,writeback=0i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=0387b682ef63fa7c3bf5d9839c0111704e4af47c3aa223da338a043c5b5e5750,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=mosquitto,com.docker.compose.version=1.26.0,container_image=eclipse-mosquitto,container_name=mosquitto,container_status=running,container_version=unknown,cpu=cpu-total,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=grafana,host=5d55d71ced7c,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=19.03.11 container_id="4cc84e7cf5961852326f7aa187ea1e2f7d08c901e445ca7affa4977cc9cb1732",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=40000000i,usage_in_usermode=40000000i,usage_percent=0.07330497607655502,usage_system=1089131880000000i,usage_total=82819573i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=0387b682ef63fa7c3bf5d9839c0111704e4af47c3aa223da338a043c5b5e5750,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=mosquitto,com.docker.compose.version=1.26.0,container_image=eclipse-mosquitto,container_name=mosquitto,container_status=running,container_version=unknown,cpu=cpu0,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=grafana,host=5d55d71ced7c,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=19.03.11 container_id="4cc84e7cf5961852326f7aa187ea1e2f7d08c901e445ca7affa4977cc9cb1732",usage_total=25231945i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=0387b682ef63fa7c3bf5d9839c0111704e4af47c3aa223da338a043c5b5e5750,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=mosquitto,com.docker.compose.version=1.26.0,container_image=eclipse-mosquitto,container_name=mosquitto,container_status=running,container_version=unknown,cpu=cpu1,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=grafana,host=5d55d71ced7c,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=19.03.11 container_id="4cc84e7cf5961852326f7aa187ea1e2f7d08c901e445ca7affa4977cc9cb1732",usage_total=617853i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=0387b682ef63fa7c3bf5d9839c0111704e4af47c3aa223da338a043c5b5e5750,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=mosquitto,com.docker.compose.version=1.26.0,container_image=eclipse-mosquitto,container_name=mosquitto,container_status=running,container_version=unknown,cpu=cpu2,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=grafana,host=5d55d71ced7c,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=19.03.11 container_id="4cc84e7cf5961852326f7aa187ea1e2f7d08c901e445ca7affa4977cc9cb1732",usage_total=44523889i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=0387b682ef63fa7c3bf5d9839c0111704e4af47c3aa223da338a043c5b5e5750,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=mosquitto,com.docker.compose.version=1.26.0,container_image=eclipse-mosquitto,container_name=mosquitto,container_status=running,container_version=unknown,cpu=cpu3,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=grafana,host=5d55d71ced7c,maintainer=Roger\ Light\ <roger@atchoo.org>,server_version=19.03.11 container_id="4cc84e7cf5961852326f7aa187ea1e2f7d08c901e445ca7affa4977cc9cb1732",usage_total=12445886i 1591889502000000000
telegraf     | > docker_container_net,com.docker.compose.config-hash=0387b682ef63fa7c3bf5d9839c0111704e4af47c3aa223da338a043c5b5e5750,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=mosquitto,com.docker.compose.version=1.26.0,container_image=eclipse-mosquitto,container_name=mosquitto,container_status=running,container_version=unknown,description=Eclipse\ Mosquitto\ MQTT\ Broker,engine_host=grafana,host=5d55d71ced7c,maintainer=Roger\ Light\ <roger@atchoo.org>,network=eth0,server_version=19.03.11 container_id="4cc84e7cf5961852326f7aa187ea1e2f7d08c901e445ca7affa4977cc9cb1732",rx_bytes=7706i,rx_dropped=0i,rx_errors=0i,rx_packets=61i,tx_bytes=1402i,tx_dropped=0i,tx_errors=0i,tx_packets=15i 1591889502000000000
telegraf     | > docker_container_status,com.docker.compose.config-hash=559b1f105a3f856dba81a07faae8c3a381bf2d11ead3a3be24c7909af925dddf,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=telegraf,com.docker.compose.version=1.26.0,container_image=telegraf,container_name=telegraf,container_status=running,container_version=unknown,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="5d55d71ced7c15b7903ef8aaa9c53910dc11f9e01461345b9084dc12b3a529ec",exitcode=0i,finished_at=1591889440168265153i,oomkilled=false,pid=21276i,started_at=1591889498163259091i,uptime_ns=4017368113i 1591889502000000000
telegraf     | > docker_container_mem,com.docker.compose.config-hash=559b1f105a3f856dba81a07faae8c3a381bf2d11ead3a3be24c7909af925dddf,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=telegraf,com.docker.compose.version=1.26.0,container_image=telegraf,container_name=telegraf,container_status=running,container_version=unknown,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 active_anon=8368128i,active_file=0i,cache=0i,container_id="5d55d71ced7c15b7903ef8aaa9c53910dc11f9e01461345b9084dc12b3a529ec",hierarchical_memory_limit=8796093018112i,inactive_anon=0i,inactive_file=3424256i,limit=4095737856i,mapped_file=0i,max_usage=13717504i,pgfault=7095i,pgmajfault=0i,pgpgin=4620i,pgpgout=1769i,rss=11538432i,rss_huge=0i,total_active_anon=8368128i,total_active_file=0i,total_cache=0i,total_inactive_anon=0i,total_inactive_file=3424256i,total_mapped_file=0i,total_pgfault=7095i,total_pgmajfault=0i,total_pgpgin=4620i,total_pgpgout=1769i,total_rss=11538432i,total_rss_huge=0i,total_unevictable=0i,total_writeback=0i,unevictable=0i,usage=13586432i,usage_percent=0.3317212301587302,writeback=0i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=559b1f105a3f856dba81a07faae8c3a381bf2d11ead3a3be24c7909af925dddf,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=telegraf,com.docker.compose.version=1.26.0,container_image=telegraf,container_name=telegraf,container_status=running,container_version=unknown,cpu=cpu-total,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="5d55d71ced7c15b7903ef8aaa9c53910dc11f9e01461345b9084dc12b3a529ec",throttling_periods=0i,throttling_throttled_periods=0i,throttling_throttled_time=0i,usage_in_kernelmode=230000000i,usage_in_usermode=420000000i,usage_percent=3.4808170334928232,usage_system=1089131920000000i,usage_total=659126078i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=559b1f105a3f856dba81a07faae8c3a381bf2d11ead3a3be24c7909af925dddf,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=telegraf,com.docker.compose.version=1.26.0,container_image=telegraf,container_name=telegraf,container_status=running,container_version=unknown,cpu=cpu0,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="5d55d71ced7c15b7903ef8aaa9c53910dc11f9e01461345b9084dc12b3a529ec",usage_total=160501374i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=559b1f105a3f856dba81a07faae8c3a381bf2d11ead3a3be24c7909af925dddf,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=telegraf,com.docker.compose.version=1.26.0,container_image=telegraf,container_name=telegraf,container_status=running,container_version=unknown,cpu=cpu1,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="5d55d71ced7c15b7903ef8aaa9c53910dc11f9e01461345b9084dc12b3a529ec",usage_total=51435884i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=559b1f105a3f856dba81a07faae8c3a381bf2d11ead3a3be24c7909af925dddf,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=telegraf,com.docker.compose.version=1.26.0,container_image=telegraf,container_name=telegraf,container_status=running,container_version=unknown,cpu=cpu2,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="5d55d71ced7c15b7903ef8aaa9c53910dc11f9e01461345b9084dc12b3a529ec",usage_total=102973341i 1591889502000000000
telegraf     | > docker_container_cpu,com.docker.compose.config-hash=559b1f105a3f856dba81a07faae8c3a381bf2d11ead3a3be24c7909af925dddf,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=telegraf,com.docker.compose.version=1.26.0,container_image=telegraf,container_name=telegraf,container_status=running,container_version=unknown,cpu=cpu3,engine_host=grafana,host=5d55d71ced7c,server_version=19.03.11 container_id="5d55d71ced7c15b7903ef8aaa9c53910dc11f9e01461345b9084dc12b3a529ec",usage_total=344215479i 1591889502000000000
telegraf     | > docker_container_net,com.docker.compose.config-hash=559b1f105a3f856dba81a07faae8c3a381bf2d11ead3a3be24c7909af925dddf,com.docker.compose.container-number=1,com.docker.compose.oneoff=False,com.docker.compose.project=docker,com.docker.compose.project.config_files=docker-compose.yml,com.docker.compose.project.working_dir=/home/pi/docker,com.docker.compose.service=telegraf,com.docker.compose.version=1.26.0,container_image=telegraf,container_name=telegraf,container_status=running,container_version=unknown,engine_host=grafana,host=5d55d71ced7c,network=eth0,server_version=19.03.11 container_id="5d55d71ced7c15b7903ef8aaa9c53910dc11f9e01461345b9084dc12b3a529ec",rx_bytes=34656i,rx_dropped=0i,rx_errors=0i,rx_packets=86i,tx_bytes=6266i,tx_dropped=0i,tx_errors=0i,tx_packets=63i 1591889502000000000
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889504855748861
influxdb     | ts=2020-06-11T15:31:45.159774Z lvl=info msg="Reading file" log_id=0NK~yfj0000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/9/_00007.wal size=10496226
influxdb     | ts=2020-06-11T15:31:49.075113Z lvl=info msg="Reading file" log_id=0NK~yfj0000 engine=tsm1 service=cacheloader path=/var/lib/influxdb/wal/_internal/monitor/9/_00008.wal size=169934
influxdb     | ts=2020-06-11T15:31:49.180274Z lvl=info msg="Opened shard" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open index_version=inmem path=/var/lib/influxdb/data/_internal/monitor/9 duration=13269.296ms
influxdb     | ts=2020-06-11T15:31:49.183071Z lvl=info msg="Open store (end)" log_id=0NK~yfj0000 service=store trace_id=0NK~ygCG000 op_name=tsdb_open op_event=end op_elapsed=13325.429ms
influxdb     | ts=2020-06-11T15:31:49.183208Z lvl=info msg="Opened service" log_id=0NK~yfj0000 service=subscriber
influxdb     | ts=2020-06-11T15:31:49.183234Z lvl=info msg="Starting monitor service" log_id=0NK~yfj0000 service=monitor
influxdb     | ts=2020-06-11T15:31:49.183255Z lvl=info msg="Registered diagnostics client" log_id=0NK~yfj0000 service=monitor name=build
influxdb     | ts=2020-06-11T15:31:49.183273Z lvl=info msg="Registered diagnostics client" log_id=0NK~yfj0000 service=monitor name=runtime
influxdb     | ts=2020-06-11T15:31:49.183290Z lvl=info msg="Registered diagnostics client" log_id=0NK~yfj0000 service=monitor name=network
influxdb     | ts=2020-06-11T15:31:49.183318Z lvl=info msg="Registered diagnostics client" log_id=0NK~yfj0000 service=monitor name=system
influxdb     | ts=2020-06-11T15:31:49.183493Z lvl=info msg="Starting precreation service" log_id=0NK~yfj0000 service=shard-precreation check_interval=10m advance_period=30m
influxdb     | ts=2020-06-11T15:31:49.183537Z lvl=info msg="Starting snapshot service" log_id=0NK~yfj0000 service=snapshot
influxdb     | ts=2020-06-11T15:31:49.183568Z lvl=info msg="Starting continuous query service" log_id=0NK~yfj0000 service=continuous_querier
influxdb     | ts=2020-06-11T15:31:49.183592Z lvl=info msg="Starting HTTP service" log_id=0NK~yfj0000 service=httpd authentication=true
influxdb     | ts=2020-06-11T15:31:49.183612Z lvl=info msg="opened HTTP access log" log_id=0NK~yfj0000 service=httpd path=stderr
influxdb     | ts=2020-06-11T15:31:49.183628Z lvl=info msg="Auth is enabled but shared-secret is blank. BearerAuthentication is disabled." log_id=0NK~yfj0000 service=httpd
influxdb     | ts=2020-06-11T15:31:49.183801Z lvl=info msg="Storing statistics" log_id=0NK~yfj0000 service=monitor db_instance=_internal db_rp=monitor interval=10s
influxdb     | ts=2020-06-11T15:31:49.183945Z lvl=info msg="Listening on HTTP" log_id=0NK~yfj0000 service=httpd addr=[::]:8086 https=false
influxdb     | ts=2020-06-11T15:31:49.184004Z lvl=info msg="Starting retention policy enforcement service" log_id=0NK~yfj0000 service=retention check_interval=30m
influxdb     | ts=2020-06-11T15:31:49.184198Z lvl=info msg="Listening for signals" log_id=0NK~yfj0000
influxdb     | ts=2020-06-11T15:31:49.487200Z lvl=info msg="Sending usage statistics to usage.influxdata.com" log_id=0NK~yfj0000
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889520125379135
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889535176968209
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889551151581051
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889564865417545
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889579865100025
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889594864488951
mosquitto    | 2020-06-11T15:33:38: Client beerfridge has exceeded timeout, disconnecting.
mosquitto    | 2020-06-11T15:34:03: New connection from 192.168.1.239 on port 1883.
mosquitto    | 2020-06-11T15:34:03: New client connected from 192.168.1.239 as beerfridge (p2, c1, k15, u'sensors').
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889662413952525
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889678438314374
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889692415390674
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889708547029015
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889722417844406
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889737627612211
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889752421906587
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889768655144009
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889783707601837
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889797737713900
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889813712422505
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889827741614123
mosquitto    | 2020-06-11T15:37:31: Client beerfridge has exceeded timeout, disconnecting.
mosquitto    | 2020-06-11T15:38:05: New connection from 192.168.1.239 on port 1883.
mosquitto    | 2020-06-11T15:38:05: New client connected from 192.168.1.239 as beerfridge (p2, c1, k15, u'sensors').
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889897890506734
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889913969644048
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889929016488936
mosquitto    | 2020-06-11T15:39:06: New connection from 192.168.1.239 on port 1883.
mosquitto    | 2020-06-11T15:39:06: Client beerfridge already connected, closing old connection.
mosquitto    | 2020-06-11T15:39:06: New client connected from 192.168.1.239 as beerfridge (p2, c1, k15, u'sensors').
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889963116636303
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591889978067160332
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.1875 1591889993120235677
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591890009197551645
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591890024148219591
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591890037286594306
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591890052310859693
telegraf     | > mqtt_home,host=5d55d71ced7c,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=24.125 1591890068284120697
mosquitto    | 2020-06-11T15:41:31: Client beerfridge has exceeded timeout, disconnecting.
danielnelson commented 4 years ago

I'm not sure then, try to reduce the the docker-compose file to the minimal case with just mosquitto and telegraf and then add the compose file and I'll try to duplicate.

tkerby commented 4 years ago

@danielnelson OK, I've got it down to mosquitto and telegraf only. Note this is running on a Raspberry Pi 4 with 4GB ram and latest Raspbian Buster.

Telegraf set to only receive from mosquito and output to a file to avoid the need for influx. Interestingly, te problem occurs slightly less often - wondering if its a timeout while processor is shared for instance? (note none of the containers are anywhere near CPU or RAM limits)

Log File:

pi@grafana:~/TelegrafBug $ docker-compose up
Creating network "telegrafbug_iot-net" with the default driver
Creating mosquitto ... done
Creating telegraf  ... done
Attaching to mosquitto, telegraf
mosquitto    | 2020-06-11T22:20:23: mosquitto version 1.6.10 starting
mosquitto    | 2020-06-11T22:20:23: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto    | 2020-06-11T22:20:23: Opening ipv4 listen socket on port 1883.
mosquitto    | 2020-06-11T22:20:23: Opening ipv6 listen socket on port 1883.
telegraf     | 2020-06-11T22:20:25Z I! Starting Telegraf 1.14.4
telegraf     | 2020-06-11T22:20:25Z I! Using config file: /etc/telegraf/telegraf.conf
telegraf     | 2020-06-11T22:20:25Z I! Loaded inputs: mqtt_consumer
telegraf     | 2020-06-11T22:20:25Z I! Loaded aggregators:
telegraf     | 2020-06-11T22:20:25Z I! Loaded processors:
telegraf     | 2020-06-11T22:20:25Z I! Loaded outputs: file
telegraf     | 2020-06-11T22:20:25Z I! Tags enabled: host=6419d72c8004
telegraf     | 2020-06-11T22:20:25Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"6419d72c8004", Flush Interval:1m0s
telegraf     | 2020-06-11T22:20:25Z D! [agent] Initializing plugins
telegraf     | 2020-06-11T22:20:25Z D! [agent] Connecting outputs
telegraf     | 2020-06-11T22:20:25Z D! [agent] Attempting connection to [outputs.file]
telegraf     | 2020-06-11T22:20:25Z D! [agent] Successfully connected to outputs.file
telegraf     | 2020-06-11T22:20:25Z D! [agent] Starting service inputs
mosquitto    | 2020-06-11T22:20:25: New connection from 192.168.48.3 on port 1883.
mosquitto    | 2020-06-11T22:20:25: New client connected from 192.168.48.3 as Telegraf-Consumer-U3nef (p2, c1, k60, u'sensors').
telegraf     | 2020-06-11T22:20:25Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
mosquitto    | 2020-06-11T22:20:32: New connection from 192.168.1.239 on port 1883.
mosquitto    | 2020-06-11T22:20:32: New client connected from 192.168.1.239 as beerfridge (p2, c1, k15, u'sensors').
mosquitto    | 2020-06-11T22:21:45: Socket error on client Telegraf-Consumer-U3nef, disconnecting.
telegraf     | 2020-06-11T22:21:45Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
telegraf     | 2020-06-11T22:21:45Z D! [inputs.mqtt_consumer] Disconnected [tcp://mosquitto:1883]
telegraf     | 2020-06-11T22:21:46Z D! [inputs.mqtt_consumer] Connecting [tcp://mosquitto:1883]
mosquitto    | 2020-06-11T22:21:46: New connection from 192.168.48.3 on port 1883.
mosquitto    | 2020-06-11T22:21:46: New client connected from 192.168.48.3 as Telegraf-Consumer-U3nef (p2, c1, k60, u'sensors').
telegraf     | 2020-06-11T22:21:46Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
telegraf     | mqtt_home,host=6419d72c8004,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=3.25 1591914049671307789
telegraf     | 2020-06-11T22:22:00Z D! [outputs.file] Wrote batch of 1 metrics in 316.346µs
telegraf     | 2020-06-11T22:22:00Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf     | 2020-06-11T22:23:00Z D! [outputs.file] Wrote batch of 1 metrics in 184.644µs
telegraf     | 2020-06-11T22:23:00Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf     | mqtt_home,host=6419d72c8004,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=3.25 1591914120005970205
telegraf     | 2020-06-11T22:23:06Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting
telegraf     | 2020-06-11T22:23:06Z D! [inputs.mqtt_consumer] Disconnected [tcp://mosquitto:1883]
mosquitto    | 2020-06-11T22:23:06: Socket error on client Telegraf-Consumer-U3nef, disconnecting.
telegraf     | 2020-06-11T22:23:07Z D! [inputs.mqtt_consumer] Connecting [tcp://mosquitto:1883]
mosquitto    | 2020-06-11T22:23:07: New connection from 192.168.48.3 on port 1883.
mosquitto    | 2020-06-11T22:23:07: New client connected from 192.168.48.3 as Telegraf-Consumer-U3nef (p2, c1, k60, u'sensors').
telegraf     | 2020-06-11T22:23:07Z I! [inputs.mqtt_consumer] Connected [tcp://mosquitto:1883]
telegraf     | mqtt_home,host=6419d72c8004,topic=home/devices/garage/beerfridge/up temperature_fridge_air_c=3.25 1591914180005873827
telegraf     | 2020-06-11T22:24:00Z D! [outputs.file] Wrote batch of 1 metrics in 238.514µs
telegraf     | 2020-06-11T22:24:00Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

Docker File


version: '3.8'
services:
    mosquitto:
        container_name: mosquitto
        restart: unless-stopped
        image: eclipse-mosquitto
        environment:
            - TZ=Europe/London
        ports:
            - '1883:1883'
            - '9001:9001'
        networks:
            - iot-net
        volumes:
            - /mnt/SSD/docker/mosquitto/config:/mosquitto/config
    telegraf:
        container_name: telegraf
        restart: unless-stopped
        image: telegraf
       # command: --test-wait 600
        networks:
            - iot-net
        environment:
            - interval=60s
            - flush_interval=60s
        depends_on:
            - mosquitto
        privileged: true
        ports:
            - 8125:8125
            - 8092:8092
            - 8094:8094
        volumes:
            - /mnt/SSD/docker/telegraftest/config:/etc/telegraf

networks:
    iot-net: null

Telegraf.cong (note mosquito username and password must be set - replaced with ** and remember to create directories in dockerfile)

# Telegraf Configuration
#
# Telegraf is entirely plugin driven. All metrics are gathered from the
# declared inputs, and sent to the declared outputs.
#
# Plugins must be declared in here to be active.
# To deactivate a plugin, comment out the name and any variables.
#
# Use 'telegraf -config telegraf.conf -test' to see what metrics a config
# file would generate.
#
# Environment variables can be used anywhere in this config file, simply surround
# them with ${}. For strings the variable must be within quotes (ie, "${STR_VAR}"),
# for numbers and booleans they should be plain (ie, ${INT_VAR}, ${BOOL_VAR})

# Global tags can be specified here in key="value" format.
[global_tags]
  # dc = "us-east-1" # will tag all metrics with dc=us-east-1
  # rack = "1a"
  ## Environment variables can be used as tags, and throughout the config file
  # user = "$USER"

# Configuration for telegraf agent
[agent]
  ## Default data collection interval for all inputs
  interval = "$interval"
  ## Rounds collection interval to 'interval'
  ## ie, if interval="10s" then always collect on :00, :10, :20, etc.
  round_interval = true

  ## Telegraf will send metrics to outputs in batches of at most
  ## metric_batch_size metrics.
  ## This controls the size of writes that Telegraf sends to output plugins.
  metric_batch_size = 1000

  ## Maximum number of unwritten metrics per output.  Increasing this value
  ## allows for longer periods of output downtime without dropping metrics at the
  ## cost of higher maximum memory usage.
  metric_buffer_limit = 10000

  ## Collection jitter is used to jitter the collection by a random amount.
  ## Each plugin will sleep for a random time within jitter before collecting.
  ## This can be used to avoid many plugins querying things like sysfs at the
  ## same time, which can have a measurable effect on the system.
  collection_jitter = "0s"

  ## Default flushing interval for all outputs. Maximum flush_interval will be
  ## flush_interval + flush_jitter
  flush_interval = "$flush_interval"
  ## Jitter the flush interval by a random amount. This is primarily to avoid
  ## large write spikes for users running a large number of telegraf instances.
  ## ie, a jitter of 5s and interval 10s means flushes will happen every 10-15s
  flush_jitter = "0s"

  ## By default or when set to "0s", precision will be set to the same
  ## timestamp order as the collection interval, with the maximum being 1s.
  ##   ie, when interval = "10s", precision will be "1s"
  ##       when interval = "250ms", precision will be "1ms"
  ## Precision will NOT be used for service inputs. It is up to each individual
  ## service input to set the timestamp at the appropriate precision.
  ## Valid time units are "ns", "us" (or "µs"), "ms", "s".
  precision = ""

  ## Log at debug level.
  # debug = false
  debug = true
  ## Log only error level messages.
  # quiet = false

  ## Log target controls the destination for logs and can be one of "file",
  ## "stderr" or, on Windows, "eventlog".  When set to "file", the output file
  ## is determined by the "logfile" setting.
  # logtarget = "file"

  ## Name of the file to be logged to when using the "file" logtarget.  If set to
  ## the empty string then logs are written to stderr.
  # logfile = ""

  ## The logfile will be rotated after the time interval specified.  When set
  ## to 0 no time based rotation is performed.  Logs are rotated only when
  ## written to, if there is no log activity rotation may be delayed.
  # logfile_rotation_interval = "0d"

  ## The logfile will be rotated when it becomes larger than the specified
  ## size.  When set to 0 no size based rotation is performed.
  # logfile_rotation_max_size = "0MB"

  ## Maximum number of rotated archives to keep, any older logs are deleted.
  ## If set to -1, no archives are removed.
  # logfile_rotation_max_archives = 5

  ## Override default hostname, if empty use os.Hostname()
  hostname = ""
  ## If set to true, do no set the "host" tag in the telegraf agent.
  omit_hostname = false

###############################################################################
#                            OUTPUT PLUGINS                                   #
###############################################################################

# # Send telegraf metrics to file(s)
[[outputs.file]]
#   ## Files to write to, "stdout" is a specially handled file.
   files = ["stdout", "/tmp/metrics.out"]
#
#   ## Use batch serialization format instead of line based delimiting.  The
#   ## batch format allows for the production of non line based output formats and
#   ## may more effiently encode metric groups.
#   # use_batch_format = false
#
#   ## The file will be rotated after the time interval specified.  When set
#   ## to 0 no time based rotation is performed.
#   # rotation_interval = "0d"
#
#   ## The logfile will be rotated when it becomes larger than the specified
#   ## size.  When set to 0 no size based rotation is performed.
#   # rotation_max_size = "0MB"
#
#   ## Maximum number of rotated archives to keep, any older logs are deleted.
#   ## If set to -1, no archives are removed.
#   # rotation_max_archives = 5
#
#   ## Data format to output.
#   ## Each data format has its own unique set of configuration options, read
#   ## more about them here:
#   ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_OUTPUT.md
#   data_format = "influx"

###############################################################################
#                            INPUT PLUGINS                                    #
###############################################################################

# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
  name_override = "mqtt_home"

  ## MQTT broker URLs to be used. The format should be scheme://host:port,
  ## schema can be tcp, ssl, or ws.
  servers = ["tcp://mosquitto:1883"]

  ## Topics that will be subscribed to.
  topics = ["home/devices/+/+/up"]

  ## The message topic will be stored in a tag specified by this value.  If set
  ## to the empty string no topic tag will be created.
  # topic_tag = "topic"

  ## QoS policy for messages
  ##   0 = at most once
  ##   1 = at least once
  ##   2 = exactly once
  ##
  ## When using a QoS of 1 or 2, you should enable persistent_session to allow
  ## resuming unacknowledged messages.
  qos = 0

  ## Connection timeout for initial connection in seconds
  connection_timeout = "60s"

  ## Maximum messages to read from the broker that have not been written by an
  ## output.  For best throughput set based on the number of metrics within
  ## each message and the size of the output's metric_batch_size.
  ##
  ## For example, if each message from the queue contains 10 metrics and the
  ## output metric_batch_size is 1000, setting this to 100 will ensure that a
  ## full batch is collected and the write is triggered immediately without
  ## waiting until the next flush_interval.
  max_undelivered_messages = 1

  ## Persistent session disables clearing of the client session on connection.
  ## In order for this option to work you must also set client_id to identity
  ## the client.  To receive messages that arrived while the client is offline,
  ## also set the qos option to 1 or 2 and don't forget to also set the QoS when
  ## publishing.
  persistent_session = false

  ## If unset, a random client ID will be generated.
  client_id = ""

  ## Username and password to connect MQTT server.
  username = "******"
  password = "******"

  ## Optional TLS Config
  # tls_ca = "/etc/telegraf/ca.pem"
  # tls_cert = "/etc/telegraf/cert.pem"
  # tls_key = "/etc/telegraf/key.pem"
  ## Use TLS but skip chain & host verification
  # insecure_skip_verify = false

  ## Data format to consume.
  ## Each data format has its own unique set of configuration options, read
  ## more about them here:
  ## https://github.com/influxdata/telegraf/blob/master/docs/DATA_FORMATS_INPUT.md
  data_format = "json"
  interval = "1s"

Mosquitto.conf (you'll need to create a mosquitto.passwd configuration file to match telegraf config and mqtt device. Remember to create directories in dockerfile)

# Config file for mosquitto
#
# See mosquitto.conf(5) for more information.
#
# Default values are shown, uncomment to change.
#
# Use the # character to indicate a comment, but only if it is the
# very first character on the line.

# =================================================================
# General configuration
# =================================================================

# Use per listener security settings.
#
# It is recommended this option be set before any other options.
#
# If this option is set to true, then all authentication and access control
# options are controlled on a per listener basis. The following options are
# affected:
#
# password_file acl_file psk_file auth_plugin auth_opt_* allow_anonymous
# auto_id_prefix allow_zero_length_clientid
#
# Note that if set to true, then a durable client (i.e. with clean session set
# to false) that has disconnected will use the ACL settings defined for the
# listener that it was most recently connected to.
#
# The default behaviour is for this to be set to false, which maintains the
# setting behaviour from previous versions of mosquitto.
#per_listener_settings false

# If a client is subscribed to multiple subscriptions that overlap, e.g. foo/#
# and foo/+/baz , then MQTT expects that when the broker receives a message on
# a topic that matches both subscriptions, such as foo/bar/baz, then the client
# should only receive the message once.
# Mosquitto keeps track of which clients a message has been sent to in order to
# meet this requirement. The allow_duplicate_messages option allows this
# behaviour to be disabled, which may be useful if you have a large number of
# clients subscribed to the same set of topics and are very concerned about
# minimising memory usage.
# It can be safely set to true if you know in advance that your clients will
# never have overlapping subscriptions, otherwise your clients must be able to
# correctly deal with duplicate messages even when then have QoS=2.
#allow_duplicate_messages false

# This option controls whether a client is allowed to connect with a zero
# length client id or not. This option only affects clients using MQTT v3.1.1
# and later. If set to false, clients connecting with a zero length client id
# are disconnected. If set to true, clients will be allocated a client id by
# the broker. This means it is only useful for clients with clean session set
# to true.
allow_zero_length_clientid true

# If allow_zero_length_clientid is true, this option allows you to set a prefix
# to automatically generated client ids to aid visibility in logs.
# Defaults to 'auto-'
auto_id_prefix auto-

# This option affects the scenario when a client subscribes to a topic that has
# retained messages. It is possible that the client that published the retained
# message to the topic had access at the time they published, but that access
# has been subsequently removed. If check_retain_source is set to true, the
# default, the source of a retained message will be checked for access rights
# before it is republished. When set to false, no check will be made and the
# retained message will always be published. This affects all listeners.
#check_retain_source true

# QoS 1 and 2 messages will be allowed inflight per client until this limit
# is exceeded.  Defaults to 0. (No maximum)
# See also max_inflight_messages
#max_inflight_bytes 0

# The maximum number of QoS 1 and 2 messages currently inflight per
# client.
# This includes messages that are partway through handshakes and
# those that are being retried. Defaults to 20. Set to 0 for no
# maximum. Setting to 1 will guarantee in-order delivery of QoS 1
# and 2 messages.
#max_inflight_messages 20

# For MQTT v5 clients, it is possible to have the server send a "server
# keepalive" value that will override the keepalive value set by the client.
# This is intended to be used as a mechanism to say that the server will
# disconnect the client earlier than it anticipated, and that the client should
# use the new keepalive value. The max_keepalive option allows you to specify
# that clients may only connect with keepalive less than or equal to this
# value, otherwise they will be sent a server keepalive telling them to use
# max_keepalive. This only applies to MQTT v5 clients. The maximum value
# allowable is 65535. Do not set below 10.
#max_keepalive 65535
#max_keepalive 2000

# For MQTT v5 clients, it is possible to have the server send a "maximum packet
# size" value that will instruct the client it will not accept MQTT packets
# with size greater than max_packet_size bytes. This applies to the full MQTT
# packet, not just the payload. Setting this option to a positive value will
# set the maximum packet size to that number of bytes. If a client sends a
# packet which is larger than this value, it will be disconnected. This applies
# to all clients regardless of the protocol version they are using, but v3.1.1
# and earlier clients will of course not have received the maximum packet size
# information. Defaults to no limit. Setting below 20 bytes is forbidden
# because it is likely to interfere with ordinary client operation, even with
# very small payloads.
#max_packet_size 0

# QoS 1 and 2 messages above those currently in-flight will be queued per
# client until this limit is exceeded.  Defaults to 0. (No maximum)
# See also max_queued_messages.
# If both max_queued_messages and max_queued_bytes are specified, packets will
# be queued until the first limit is reached.
#max_queued_bytes 0

# The maximum number of QoS 1 and 2 messages to hold in a queue per client
# above those that are currently in-flight.  Defaults to 100. Set
# to 0 for no maximum (not recommended).
# See also queue_qos0_messages.
# See also max_queued_bytes.
#max_queued_messages 100
#
# This option sets the maximum number of heap memory bytes that the broker will
# allocate, and hence sets a hard limit on memory use by the broker.  Memory
# requests that exceed this value will be denied. The effect will vary
# depending on what has been denied. If an incoming message is being processed,
# then the message will be dropped and the publishing client will be
# disconnected. If an outgoing message is being sent, then the individual
# message will be dropped and the receiving client will be disconnected.
# Defaults to no limit.
#memory_limit 0

# This option sets the maximum publish payload size that the broker will allow.
# Received messages that exceed this size will not be accepted by the broker.
# The default value is 0, which means that all valid MQTT messages are
# accepted. MQTT imposes a maximum payload size of 268435455 bytes.
#message_size_limit 0

# This option allows persistent clients (those with clean session set to false)
# to be removed if they do not reconnect within a certain time frame.
#
# This is a non-standard option in MQTT V3.1 but allowed in MQTT v3.1.1.
#
# Badly designed clients may set clean session to false whilst using a randomly
# generated client id. This leads to persistent clients that will never
# reconnect. This option allows these clients to be removed.
#
# The expiration period should be an integer followed by one of h d w m y for
# hour, day, week, month and year respectively. For example
#
# persistent_client_expiration 2m
# persistent_client_expiration 14d
# persistent_client_expiration 1y
#
# The default if not set is to never expire persistent clients.
#persistent_client_expiration

# Write process id to a file. Default is a blank string which means
# a pid file shouldn't be written.
# This should be set to /var/run/mosquitto.pid if mosquitto is
# being run automatically on boot with an init script and
# start-stop-daemon or similar.
#pid_file

# Set to true to queue messages with QoS 0 when a persistent client is
# disconnected. These messages are included in the limit imposed by
# max_queued_messages and max_queued_bytes
# Defaults to false.
# This is a non-standard option for the MQTT v3.1 spec but is allowed in
# v3.1.1.
#queue_qos0_messages false

# Set to false to disable retained message support. If a client publishes a
# message with the retain bit set, it will be disconnected if this is set to
# false.
#retain_available true

# Disable Nagle's algorithm on client sockets. This has the effect of reducing
# latency of individual messages at the potential cost of increasing the number
# of packets being sent.
#set_tcp_nodelay false

# Time in seconds between updates of the $SYS tree.
# Set to 0 to disable the publishing of the $SYS tree.
#sys_interval 10

# The MQTT specification requires that the QoS of a message delivered to a
# subscriber is never upgraded to match the QoS of the subscription. Enabling
# this option changes this behaviour. If upgrade_outgoing_qos is set true,
# messages sent to a subscriber will always match the QoS of its subscription.
# This is a non-standard option explicitly disallowed by the spec.
#upgrade_outgoing_qos false

# When run as root, drop privileges to this user and its primary
# group.
# Set to root to stay as root, but this is not recommended.
# If run as a non-root user, this setting has no effect.
# Note that on Windows this has no effect and so mosquitto should
# be started by the user you wish it to run as.
user mosquitto

# =================================================================
# Default listener
# =================================================================

# IP address/hostname to bind the default listener to. If not
# given, the default listener will not be bound to a specific
# address and so will be accessible to all network interfaces.
# bind_address ip-address/host name
#bind_address

# Port to use for the default listener.
port 1883

# Bind the listener to a specific interface. This is similar to
# bind_address above but is useful when an interface has multiple addresses or
# the address may change. It is valid to use this with the bind_address option,
# but take care that the interface you are binding to contains the address you
# are binding to, otherwise you will not be able to connect.
# Example: bind_interface eth0
#bind_interface

# When a listener is using the websockets protocol, it is possible to serve
# http data as well. Set http_dir to a directory which contains the files you
# wish to serve. If this option is not specified, then no normal http
# connections will be possible.
#http_dir

# The maximum number of client connections to allow. This is
# a per listener setting.
# Default is -1, which means unlimited connections.
# Note that other process limits mean that unlimited connections
# are not really possible. Typically the default maximum number of
# connections possible is around 1024.
max_connections -1

# Choose the protocol to use when listening.
# This can be either mqtt or websockets.
# Websockets support is currently disabled by default at compile time.
# Certificate based TLS may be used with websockets, except that
# only the cafile, certfile, keyfile and ciphers options are supported.
#protocol mqtt

# Set use_username_as_clientid to true to replace the clientid that a client
# connected with with its username. This allows authentication to be tied to
# the clientid, which means that it is possible to prevent one client
# disconnecting another by using the same clientid.
# If a client connects with no username it will be disconnected as not
# authorised when this option is set to true.
# Do not use in conjunction with clientid_prefixes.
# See also use_identity_as_username.
#use_username_as_clientid

# -----------------------------------------------------------------
# Certificate based SSL/TLS support
# -----------------------------------------------------------------
# The following options can be used to enable SSL/TLS support for
# this listener. Note that the recommended port for MQTT over TLS
# is 8883, but this must be set manually.
#
# See also the mosquitto-tls man page.

# At least one of cafile or capath must be defined. They both
# define methods of accessing the PEM encoded Certificate
# Authority certificates that have signed your server certificate
# and that you wish to trust.
# cafile defines the path to a file containing the CA certificates.
# capath defines a directory that will be searched for files
# containing the CA certificates. For capath to work correctly, the
# certificate files must have ".crt" as the file ending and you must run
# "openssl rehash <path to capath>" each time you add/remove a certificate.
#cafile
#capath

# Path to the PEM encoded server certificate.
#certfile

# Path to the PEM encoded keyfile.
#keyfile

# If you have require_certificate set to true, you can create a certificate
# revocation list file to revoke access to particular client certificates. If
# you have done this, use crlfile to point to the PEM encoded revocation file.
#crlfile

# If you wish to control which encryption ciphers are used, use the ciphers
# option. The list of available ciphers can be obtained using the "openssl
# ciphers" command and should be provided in the same format as the output of
# that command.
# If unset defaults to DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:@STRENGTH
#ciphers DEFAULT:!aNULL:!eNULL:!LOW:!EXPORT:!SSLv2:@STRENGTH

# To allow the use of ephemeral DH key exchange, which provides forward
# security, the listener must load DH parameters. This can be specified with
# the dhparamfile option. The dhparamfile can be generated with the command
# e.g. "openssl dhparam -out dhparam.pem 2048"
#dhparamfile

# By default a TLS enabled listener will operate in a similar fashion to a
# https enabled web server, in that the server has a certificate signed by a CA
# and the client will verify that it is a trusted certificate. The overall aim
# is encryption of the network traffic. By setting require_certificate to true,
# the client must provide a valid certificate in order for the network
# connection to proceed. This allows access to the broker to be controlled
# outside of the mechanisms provided by MQTT.
#require_certificate false

# This option defines the version of the TLS protocol to use for this listener.
# The default value allows all of v1.3, v1.2 and v1.1. The valid values are
# tlsv1.3 tlsv1.2 and tlsv1.1.
#tls_version

# If require_certificate is true, you may set use_identity_as_username to true
# to use the CN value from the client certificate as a username. If this is
# true, the password_file option will not be used for this listener.
# This takes priority over use_subject_as_username.
# See also use_subject_as_username.
#use_identity_as_username false

# If require_certificate is true, you may set use_subject_as_username to true
# to use the complete subject value from the client certificate as a username.
# If this is true, the password_file option will not be used for this listener.
# See also use_identity_as_username
#use_subject_as_username false

# -----------------------------------------------------------------
# Pre-shared-key based SSL/TLS support
# -----------------------------------------------------------------
# The following options can be used to enable PSK based SSL/TLS support for
# this listener. Note that the recommended port for MQTT over TLS is 8883, but
# this must be set manually.
#
# See also the mosquitto-tls man page and the "Certificate based SSL/TLS
# support" section. Only one of certificate or PSK encryption support can be
# enabled for any listener.

# The psk_hint option enables pre-shared-key support for this listener and also
# acts as an identifier for this listener. The hint is sent to clients and may
# be used locally to aid authentication. The hint is a free form string that
# doesn't have much meaning in itself, so feel free to be creative.
# If this option is provided, see psk_file to define the pre-shared keys to be
# used or create a security plugin to handle them.
#psk_hint

# When using PSK, the encryption ciphers used will be chosen from the list of
# available PSK ciphers. If you want to control which ciphers are available,
# use the "ciphers" option.  The list of available ciphers can be obtained
# using the "openssl ciphers" command and should be provided in the same format
# as the output of that command.
#ciphers

# Set use_identity_as_username to have the psk identity sent by the client used
# as its username. Authentication will be carried out using the PSK rather than
# the MQTT username/password and so password_file will not be used for this
# listener.
#use_identity_as_username false

# =================================================================
# Extra listeners
# =================================================================

# Listen on a port/ip address combination. By using this variable
# multiple times, mosquitto can listen on more than one port. If
# this variable is used and neither bind_address nor port given,
# then the default listener will not be started.
# The port number to listen on must be given. Optionally, an ip
# address or host name may be supplied as a second argument. In
# this case, mosquitto will attempt to bind the listener to that
# address and so restrict access to the associated network and
# interface. By default, mosquitto will listen on all interfaces.
# Note that for a websockets listener it is not possible to bind to a host
# name.
# listener port-number [ip address/host name]
#listener

# Bind the listener to a specific interface. This is similar to
# the [ip address/host name] part of the listener definition, but is useful
# when an interface has multiple addresses or the address may change. It is
# valid to use this with the [ip address/host name] part of the listener
# definition, but take care that the interface you are binding to contains the
# address you are binding to, otherwise you will not be able to connect.
# Only available on Linux and requires elevated privileges.
#
# Example: bind_interface eth0
#bind_interface

# When a listener is using the websockets protocol, it is possible to serve
# http data as well. Set http_dir to a directory which contains the files you
# wish to serve. If this option is not specified, then no normal http
# connections will be possible.
#http_dir

# The maximum number of client connections to allow. This is
# a per listener setting.
# Default is -1, which means unlimited connections.
# Note that other process limits mean that unlimited connections
# are not really possible. Typically the default maximum number of
# connections possible is around 1024.
#max_connections -1

# The listener can be restricted to operating within a topic hierarchy using
# the mount_point option. This is achieved be prefixing the mount_point string
# to all topics for any clients connected to this listener. This prefixing only
# happens internally to the broker; the client will not see the prefix.
#mount_point

# Choose the protocol to use when listening.
# This can be either mqtt or websockets.
# Certificate based TLS may be used with websockets, except that only the
# cafile, certfile, keyfile and ciphers options are supported.
#protocol mqtt

# Set use_username_as_clientid to true to replace the clientid that a client
# connected with with its username. This allows authentication to be tied to
# the clientid, which means that it is possible to prevent one client
# disconnecting another by using the same clientid.
# If a client connects with no username it will be disconnected as not
# authorised when this option is set to true.
# Do not use in conjunction with clientid_prefixes.
# See also use_identity_as_username.
#use_username_as_clientid

# Change the websockets headers size. This is a global option, it is not
# possible to set per listener. This option sets the size of the buffer used in
# the libwebsockets library when reading HTTP headers. If you are passing large
# header data such as cookies then you may need to increase this value. If left
# unset, or set to 0, then the default of 1024 bytes will be used.
#websockets_headers_size

# -----------------------------------------------------------------
# Certificate based SSL/TLS support
# -----------------------------------------------------------------
# The following options can be used to enable certificate based SSL/TLS support
# for this listener. Note that the recommended port for MQTT over TLS is 8883,
# but this must be set manually.
#
# See also the mosquitto-tls man page and the "Pre-shared-key based SSL/TLS
# support" section. Only one of certificate or PSK encryption support can be
# enabled for any listener.

# At least one of cafile or capath must be defined to enable certificate based
# TLS encryption. They both define methods of accessing the PEM encoded
# Certificate Authority certificates that have signed your server certificate
# and that you wish to trust.
# cafile defines the path to a file containing the CA certificates.
# capath defines a directory that will be searched for files
# containing the CA certificates. For capath to work correctly, the
# certificate files must have ".crt" as the file ending and you must run
# "openssl rehash <path to capath>" each time you add/remove a certificate.
#cafile
#capath

# Path to the PEM encoded server certificate.
#certfile

# Path to the PEM encoded keyfile.
#keyfile

# If you wish to control which encryption ciphers are used, use the ciphers
# option. The list of available ciphers can be optained using the "openssl
# ciphers" command and should be provided in the same format as the output of
# that command.
#ciphers

# If you have require_certificate set to true, you can create a certificate
# revocation list file to revoke access to particular client certificates. If
# you have done this, use crlfile to point to the PEM encoded revocation file.
#crlfile

# To allow the use of ephemeral DH key exchange, which provides forward
# security, the listener must load DH parameters. This can be specified with
# the dhparamfile option. The dhparamfile can be generated with the command
# e.g. "openssl dhparam -out dhparam.pem 2048"
#dhparamfile

# By default an TLS enabled listener will operate in a similar fashion to a
# https enabled web server, in that the server has a certificate signed by a CA
# and the client will verify that it is a trusted certificate. The overall aim
# is encryption of the network traffic. By setting require_certificate to true,
# the client must provide a valid certificate in order for the network
# connection to proceed. This allows access to the broker to be controlled
# outside of the mechanisms provided by MQTT.
#require_certificate false

# If require_certificate is true, you may set use_identity_as_username to true
# to use the CN value from the client certificate as a username. If this is
# true, the password_file option will not be used for this listener.
#use_identity_as_username false

# -----------------------------------------------------------------
# Pre-shared-key based SSL/TLS support
# -----------------------------------------------------------------
# The following options can be used to enable PSK based SSL/TLS support for
# this listener. Note that the recommended port for MQTT over TLS is 8883, but
# this must be set manually.
#
# See also the mosquitto-tls man page and the "Certificate based SSL/TLS
# support" section. Only one of certificate or PSK encryption support can be
# enabled for any listener.

# The psk_hint option enables pre-shared-key support for this listener and also
# acts as an identifier for this listener. The hint is sent to clients and may
# be used locally to aid authentication. The hint is a free form string that
# doesn't have much meaning in itself, so feel free to be creative.
# If this option is provided, see psk_file to define the pre-shared keys to be
# used or create a security plugin to handle them.
#psk_hint

# When using PSK, the encryption ciphers used will be chosen from the list of
# available PSK ciphers. If you want to control which ciphers are available,
# use the "ciphers" option.  The list of available ciphers can be optained
# using the "openssl ciphers" command and should be provided in the same format
# as the output of that command.
#ciphers

# Set use_identity_as_username to have the psk identity sent by the client used
# as its username. Authentication will be carried out using the PSK rather than
# the MQTT username/password and so password_file will not be used for this
# listener.
#use_identity_as_username false

# =================================================================
# Persistence
# =================================================================

# If persistence is enabled, save the in-memory database to disk
# every autosave_interval seconds. If set to 0, the persistence
# database will only be written when mosquitto exits. See also
# autosave_on_changes.
# Note that writing of the persistence database can be forced by
# sending mosquitto a SIGUSR1 signal.
autosave_interval 1800

# If true, mosquitto will count the number of subscription changes, retained
# messages received and queued messages and if the total exceeds
# autosave_interval then the in-memory database will be saved to disk.
# If false, mosquitto will save the in-memory database to disk by treating
# autosave_interval as a time in seconds.
#autosave_on_changes false

# Save persistent message data to disk (true/false).
# This saves information about all messages, including
# subscriptions, currently in-flight messages and retained
# messages.
# retained_persistence is a synonym for this option.
persistence true

# The filename to use for the persistent database, not including
# the path.
persistence_file mosquitto.db

# Location for persistent database. Must include trailing /
# Default is an empty string (current directory).
# Set to e.g. /var/lib/mosquitto/ if running as a proper service on Linux or
# similar.
persistence_location /mosquitto/data/

# =================================================================
# Logging
# =================================================================

# Places to log to. Use multiple log_dest lines for multiple
# logging destinations.
# Possible destinations are: stdout stderr syslog topic file
#
# stdout and stderr log to the console on the named output.
#
# syslog uses the userspace syslog facility which usually ends up
# in /var/log/messages or similar.
#
# topic logs to the broker topic '$SYS/broker/log/<severity>',
# where severity is one of D, E, W, N, I, M which are debug, error,
# warning, notice, information and message. Message type severity is used by
# the subscribe/unsubscribe log_types and publishes log messages to
# $SYS/broker/log/M/susbcribe or $SYS/broker/log/M/unsubscribe.
#
# The file destination requires an additional parameter which is the file to be
# logged to, e.g. "log_dest file /var/log/mosquitto.log". The file will be
# closed and reopened when the broker receives a HUP signal. Only a single file
# destination may be configured.
#
# Note that if the broker is running as a Windows service it will default to
# "log_dest none" and neither stdout nor stderr logging is available.
# Use "log_dest none" if you wish to disable logging.
log_dest file /mosquitto/log/mosquitto.log
log_dest stdout

# Types of messages to log. Use multiple log_type lines for logging
# multiple types of messages.
# Possible types are: debug, error, warning, notice, information,
# none, subscribe, unsubscribe, websockets, all.
# Note that debug type messages are for decoding the incoming/outgoing
# network packets. They are not logged in "topics".
log_type error
log_type warning
log_type notice
log_type information

# If set to true, client connection and disconnection messages will be included
# in the log.
connection_messages true

# If using syslog logging (not on Windows), messages will be logged to the
# "daemon" facility by default. Use the log_facility option to choose which of
# local0 to local7 to log to instead. The option value should be an integer
# value, e.g. "log_facility 5" to use local5.
#log_facility

# If set to true, add a timestamp value to each log message.
log_timestamp true

# Set the format of the log timestamp. If left unset, this is the number of
# seconds since the Unix epoch.
# This is a free text string which will be passed to the strftime function. To
# get an ISO 8601 datetime, for example:
log_timestamp_format %Y-%m-%dT%H:%M:%S
#log_timestamp_format

# Change the websockets logging level. This is a global option, it is not
# possible to set per listener. This is an integer that is interpreted by
# libwebsockets as a bit mask for its lws_log_levels enum. See the
# libwebsockets documentation for more details. "log_type websockets" must also
# be enabled.
#websockets_log_level 0

# =================================================================
# Security
# =================================================================

# If set, only clients that have a matching prefix on their
# clientid will be allowed to connect to the broker. By default,
# all clients may connect.
# For example, setting "secure-" here would mean a client "secure-
# client" could connect but another with clientid "mqtt" couldn't.
#clientid_prefixes

# Boolean value that determines whether clients that connect
# without providing a username are allowed to connect. If set to
# false then a password file should be created (see the
# password_file option) to control authenticated client access.
#
# Defaults to true if no other security options are set. If `password_file` or
# `psk_file` is set, or if an authentication plugin is loaded which implements
# username/password or TLS-PSK checks, then `allow_anonymous` defaults to
# false.
#
allow_anonymous false

# -----------------------------------------------------------------
# Default authentication and topic access control
# -----------------------------------------------------------------

# Control access to the broker using a password file. This file can be
# generated using the mosquitto_passwd utility. If TLS support is not compiled
# into mosquitto (it is recommended that TLS support should be included) then
# plain text passwords are used, in which case the file should be a text file
# with lines in the format:
# username:password
# The password (and colon) may be omitted if desired, although this
# offers very little in the way of security.
#
# See the TLS client require_certificate and use_identity_as_username options
# for alternative authentication options. If an auth_plugin is used as well as
# password_file, the auth_plugin check will be made first.
password_file /mosquitto/config/mosquitto.password

# Access may also be controlled using a pre-shared-key file. This requires
# TLS-PSK support and a listener configured to use it. The file should be text
# lines in the format:
# identity:key
# The key should be in hexadecimal format without a leading "0x".
# If an auth_plugin is used as well, the auth_plugin check will be made first.
#psk_file

# Control access to topics on the broker using an access control list
# file. If this parameter is defined then only the topics listed will
# have access.
# If the first character of a line of the ACL file is a # it is treated as a
# comment.
# Topic access is added with lines of the format:
#
# topic [read|write|readwrite] <topic>
#
# The access type is controlled using "read", "write" or "readwrite". This
# parameter is optional (unless <topic> contains a space character) - if not
# given then the access is read/write.  <topic> can contain the + or #
# wildcards as in subscriptions.
#
# The first set of topics are applied to anonymous clients, assuming
# allow_anonymous is true. User specific topic ACLs are added after a
# user line as follows:
#
# user <username>
#
# The username referred to here is the same as in password_file. It is
# not the clientid.
#
#
# If is also possible to define ACLs based on pattern substitution within the
# topic. The patterns available for substition are:
#
# %c to match the client id of the client
# %u to match the username of the client
#
# The substitution pattern must be the only text for that level of hierarchy.
#
# The form is the same as for the topic keyword, but using pattern as the
# keyword.
# Pattern ACLs apply to all users even if the "user" keyword has previously
# been given.
#
# If using bridges with usernames and ACLs, connection messages can be allowed
# with the following pattern:
# pattern write $SYS/broker/connection/%c/state
#
# pattern [read|write|readwrite] <topic>
#
# Example:
#
# pattern write sensor/%u/data
#
# If an auth_plugin is used as well as acl_file, the auth_plugin check will be
# made first.
#acl_file

# -----------------------------------------------------------------
# External authentication and topic access plugin options
# -----------------------------------------------------------------

# External authentication and access control can be supported with the
# auth_plugin option. This is a path to a loadable plugin. See also the
# auth_opt_* options described below.
#
# The auth_plugin option can be specified multiple times to load multiple
# plugins. The plugins will be processed in the order that they are specified
# here. If the auth_plugin option is specified alongside either of
# password_file or acl_file then the plugin checks will be made first.
#
#auth_plugin

# If the auth_plugin option above is used, define options to pass to the
# plugin here as described by the plugin instructions. All options named
# using the format auth_opt_* will be passed to the plugin, for example:
#
# auth_opt_db_host
# auth_opt_db_port
# auth_opt_db_username
# auth_opt_db_password

# =================================================================
# Bridges
# =================================================================

# A bridge is a way of connecting multiple MQTT brokers together.
# Create a new bridge using the "connection" option as described below. Set
# options for the bridges using the remaining parameters. You must specify the
# address and at least one topic to subscribe to.
#
# Each connection must have a unique name.
#
# The address line may have multiple host address and ports specified. See
# below in the round_robin description for more details on bridge behaviour if
# multiple addresses are used. Note that if you use an IPv6 address, then you
# are required to specify a port.
#
# The direction that the topic will be shared can be chosen by
# specifying out, in or both, where the default value is out.
# The QoS level of the bridged communication can be specified with the next
# topic option. The default QoS level is 0, to change the QoS the topic
# direction must also be given.
#
# The local and remote prefix options allow a topic to be remapped when it is
# bridged to/from the remote broker. This provides the ability to place a topic
# tree in an appropriate location.
#
# For more details see the mosquitto.conf man page.
#
# Multiple topics can be specified per connection, but be careful
# not to create any loops.
#
# If you are using bridges with cleansession set to false (the default), then
# you may get unexpected behaviour from incoming topics if you change what
# topics you are subscribing to. This is because the remote broker keeps the
# subscription for the old topic. If you have this problem, connect your bridge
# with cleansession set to true, then reconnect with cleansession set to false
# as normal.
#connection <name>
#address <host>[:<port>] [<host>[:<port>]]
#topic <topic> [[[out | in | both] qos-level] local-prefix remote-prefix]

# If a bridge has topics that have "out" direction, the default behaviour is to
# send an unsubscribe request to the remote broker on that topic. This means
# that changing a topic direction from "in" to "out" will not keep receiving
# incoming messages. Sending these unsubscribe requests is not always
# desirable, setting bridge_attempt_unsubscribe to false will disable sending
# the unsubscribe request.
#bridge_attempt_unsubscribe true

# Set the version of the MQTT protocol to use with for this bridge. Can be one
# of mqttv311 or mqttv11. Defaults to mqttv311.
#bridge_protocol_version mqttv311

# Set the clean session variable for this bridge.
# When set to true, when the bridge disconnects for any reason, all
# messages and subscriptions will be cleaned up on the remote
# broker. Note that with cleansession set to true, there may be a
# significant amount of retained messages sent when the bridge
# reconnects after losing its connection.
# When set to false, the subscriptions and messages are kept on the
# remote broker, and delivered when the bridge reconnects.
#cleansession false

# Set the amount of time a bridge using the lazy start type must be idle before
# it will be stopped. Defaults to 60 seconds.
#idle_timeout 60

# Set the keepalive interval for this bridge connection, in
# seconds.
#keepalive_interval 60

# Set the clientid to use on the local broker. If not defined, this defaults to
# 'local.<clientid>'. If you are bridging a broker to itself, it is important
# that local_clientid and clientid do not match.
#local_clientid

# If set to true, publish notification messages to the local and remote brokers
# giving information about the state of the bridge connection. Retained
# messages are published to the topic $SYS/broker/connection/<clientid>/state
# unless the notification_topic option is used.
# If the message is 1 then the connection is active, or 0 if the connection has
# failed.
# This uses the last will and testament feature.
#notifications true

# Choose the topic on which notification messages for this bridge are
# published. If not set, messages are published on the topic
# $SYS/broker/connection/<clientid>/state
#notification_topic

# Set the client id to use on the remote end of this bridge connection. If not
# defined, this defaults to 'name.hostname' where name is the connection name
# and hostname is the hostname of this computer.
# This replaces the old "clientid" option to avoid confusion. "clientid"
# remains valid for the time being.
#remote_clientid

# Set the password to use when connecting to a broker that requires
# authentication. This option is only used if remote_username is also set.
# This replaces the old "password" option to avoid confusion. "password"
# remains valid for the time being.
#remote_password

# Set the username to use when connecting to a broker that requires
# authentication.
# This replaces the old "username" option to avoid confusion. "username"
# remains valid for the time being.
#remote_username

# Set the amount of time a bridge using the automatic start type will wait
# until attempting to reconnect.
# This option can be configured to use a constant delay time in seconds, or to
# use a backoff mechanism based on "Decorrelated Jitter", which adds a degree
# of randomness to when the restart occurs.
#
# Set a constant timeout of 20 seconds:
# restart_timeout 20
#
# Set backoff with a base (start value) of 10 seconds and a cap (upper limit) of
# 60 seconds:
# restart_timeout 10 30
#
# Defaults to jitter with a base of 5 and cap of 30
#restart_timeout 5 30

# If the bridge has more than one address given in the address/addresses
# configuration, the round_robin option defines the behaviour of the bridge on
# a failure of the bridge connection. If round_robin is false, the default
# value, then the first address is treated as the main bridge connection. If
# the connection fails, the other secondary addresses will be attempted in
# turn. Whilst connected to a secondary bridge, the bridge will periodically
# attempt to reconnect to the main bridge until successful.
# If round_robin is true, then all addresses are treated as equals. If a
# connection fails, the next address will be tried and if successful will
# remain connected until it fails
#round_robin false

# Set the start type of the bridge. This controls how the bridge starts and
# can be one of three types: automatic, lazy and once. Note that RSMB provides
# a fourth start type "manual" which isn't currently supported by mosquitto.
#
# "automatic" is the default start type and means that the bridge connection
# will be started automatically when the broker starts and also restarted
# after a short delay (30 seconds) if the connection fails.
#
# Bridges using the "lazy" start type will be started automatically when the
# number of queued messages exceeds the number set with the "threshold"
# parameter. It will be stopped automatically after the time set by the
# "idle_timeout" parameter. Use this start type if you wish the connection to
# only be active when it is needed.
#
# A bridge using the "once" start type will be started automatically when the
# broker starts but will not be restarted if the connection fails.
#start_type automatic

# Set the number of messages that need to be queued for a bridge with lazy
# start type to be restarted. Defaults to 10 messages.
# Must be less than max_queued_messages.
#threshold 10

# If try_private is set to true, the bridge will attempt to indicate to the
# remote broker that it is a bridge not an ordinary client. If successful, this
# means that loop detection will be more effective and that retained messages
# will be propagated correctly. Not all brokers support this feature so it may
# be necessary to set try_private to false if your bridge does not connect
# properly.
#try_private true

# -----------------------------------------------------------------
# Certificate based SSL/TLS support
# -----------------------------------------------------------------
# Either bridge_cafile or bridge_capath must be defined to enable TLS support
# for this bridge.
# bridge_cafile defines the path to a file containing the
# Certificate Authority certificates that have signed the remote broker
# certificate.
# bridge_capath defines a directory that will be searched for files containing
# the CA certificates. For bridge_capath to work correctly, the certificate
# files must have ".crt" as the file ending and you must run "openssl rehash
# <path to capath>" each time you add/remove a certificate.
#bridge_cafile
#bridge_capath

# If the remote broker has more than one protocol available on its port, e.g.
# MQTT and WebSockets, then use bridge_alpn to configure which protocol is
# requested. Note that WebSockets support for bridges is not yet available.
#bridge_alpn

# When using certificate based encryption, bridge_insecure disables
# verification of the server hostname in the server certificate. This can be
# useful when testing initial server configurations, but makes it possible for
# a malicious third party to impersonate your server through DNS spoofing, for
# example. Use this option in testing only. If you need to resort to using this
# option in a production environment, your setup is at fault and there is no
# point using encryption.
#bridge_insecure false

# Path to the PEM encoded client certificate, if required by the remote broker.
#bridge_certfile

# Path to the PEM encoded client private key, if required by the remote broker.
#bridge_keyfile

# -----------------------------------------------------------------
# PSK based SSL/TLS support
# -----------------------------------------------------------------
# Pre-shared-key encryption provides an alternative to certificate based
# encryption. A bridge can be configured to use PSK with the bridge_identity
# and bridge_psk options. These are the client PSK identity, and pre-shared-key
# in hexadecimal format with no "0x". Only one of certificate and PSK based
# encryption can be used on one
# bridge at once.
#bridge_identity
#bridge_psk

# =================================================================
# External config files
# =================================================================

# External configuration files may be included by using the
# include_dir option. This defines a directory that will be searched
# for config files. All files that end in '.conf' will be loaded as
# a configuration file. It is best to have this as the last option
# in the main file. This option will only be processed from the main
# configuration file. The directory specified must not contain the
# main configuration file.
# Files within include_dir will be loaded sorted in case-sensitive
# alphabetical order, with capital letters ordered first. If this option is
# given multiple times, all of the files from the first instance will be
# processed before the next instance. See the man page for examples.
#include_dir

Sample IoT Device Data

home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.4375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.4375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.4375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.4375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.4375}
home/devices/garage/beerfridge/up {"temperature_fridge_air_c":4.4375}
tkerby commented 4 years ago

To follow up a little more I rebuilt the setup on a different host, this time using docker running on a Synology NAS.

No problems seen connecting to mqtt at first - it maintained the server connection just fine. I then reprogrammed my thermometer sensor to the new mqtt host and started sending data. Immediately got pingresp problems as before.

So it seems that the error relates somehow to data being sent or consumed - probably related to the mqtt data from the sensor being sent after the pingresp we saw in the network logs

I’m wondering - could the sensor be publishing too quickly and telegraf is seeing the data before it thinks it’s got a pingresp or something similar?

tkerby commented 4 years ago

@danielnelson Did you see the response on https://github.com/eclipse/paho.mqtt.golang/issues/430 ?

I'm getting some data through but I suspect something is blocking while telegraf processes that data causing the pingresp error. Its probably why I see it less often when sending to a file.

Whats your thoughts on how to configure the flush intervals and QOS settings for best performance to avoid this?

danielnelson commented 4 years ago

Try increasing max_undelivered_messages in the mqtt_consumer to be a bit larger than your agent metric_batch_size, and reduce the flush_interval. This will ensure that when MQTT is busy the output will always have enough data to send immediately, and lower the maximum time between writes if there are fewer messages.

[[agent]]
  flush_interval = "5s"
  metric_batch_size = 1000

[[inputs.mqtt_consumer]]
  max_undelivered_messages = 1500
SoulRaven commented 4 years ago

same problem here with VerneMQ and HAProxy in cluster mode:

2020-08-27T18:56:24Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: EOF 2020-08-27T18:57:01Z E! [inputs.snmp] Error in plugin: agent PGMO-01-PRT-XXX-PRO-01:161: gathering table prtCoverTable: performing bulk walk for field prtCoverStatusPrinterCover: Request timeout (after 3 retries) 2020-08-27T18:57:41Z E! [inputs.snmp] Error in plugin: agent PGMO-01-PRT-XXX-PRO-01:161: gathering table prtCoverTable: performing bulk walk for field prtCoverDescription: Request timeout (after 3 retries) 2020-08-27T18:58:21Z E! [inputs.snmp] Error in plugin: agent PGMO-01-PRT-XXX-PRO-01:161: gathering table prtMarkerSuppliesTable: performing bulk walk for field prtMarkerSuppliesDescription: Request timeout (after 3 retries) 2020-08-27T19:00:00Z I! [inputs.mqtt_consumer] Connected [tcp://10.20.13.228:1883] 2020-08-27T19:00:40Z E! [inputs.snmp] Error in plugin: agent PGMO-01-PRT-XXX-PRO-01:161: performing get on field sysName: Request timeout (after 3 retries) 2020-08-27T19:01:20Z E! [inputs.snmp] Error in plugin: agent PGMO-01-PRT-XXX-PRO-01:161: gathering table hrPrinterTable: performing bulk walk for field hrPrinterStatus: Request timeout (after 3 retries) 2020-08-27T19:01:24Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: EOF

MattBrittan commented 3 years ago

Note that I have just released v1.3.0 of github.com/eclipse/paho.mqtt.golang and will be interested to see if moving to that fixes the issue - there have been considerable changes since the last release (18 months ago). If it does not I would be interested to see logs with the paho.mqtt.golang logging enabled because this would help confirm where the issue lies.

takenalias commented 3 years ago

So I have also been seeing an issue with using Telegraf with The Things Network Broker. I suspect it is related

`Jan 09 11:42:05 adelvps telegraf[2490]: 2021-01-09T00:42:05Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: EOF Jan 09 11:42:05 adelvps telegraf[2490]: 2021-01-09T00:42:05Z D! [inputs.mqtt_consumer] Disconnected [tcp://router.au.thethings.network:1883] Jan 09 11:42:05 adelvps telegraf[2490]: 2021-01-09T00:42:05Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: EOF Jan 09 11:42:05 adelvps telegraf[2490]: 2021-01-09T00:42:05Z D! [inputs.mqtt_consumer] Disconnected [tcp://router.au.thethings.network:1883] Jan 09 11:42:10 adelvps telegraf[2490]: 2021-01-09T00:42:10Z D! [inputs.mqtt_consumer] Connecting [tcp://router.au.thethings.network:1883] Jan 09 11:42:10 adelvps telegraf[2490]: 2021-01-09T00:42:10Z D! [inputs.mqtt_consumer] Connecting [tcp://router.au.thethings.network:1883] Jan 09 11:42:10 adelvps telegraf[2490]: 2021-01-09T00:42:10Z I! [inputs.mqtt_consumer] Connected [tcp://router.au.thethings.network:1883] Jan 09 11:42:10 adelvps telegraf[2490]: 2021-01-09T00:42:10Z D! [outputs.influxdb] Wrote batch of 14 metrics in 7.8581ms Jan 09 11:42:10 adelvps telegraf[2490]: 2021-01-09T00:42:10Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics Jan 09 11:42:10 adelvps telegraf[2490]: 2021-01-09T00:42:10Z I! [inputs.mqtt_consumer] Connected [tcp://router.au.thethings.network:1883] Jan 09 11:42:15 adelvps telegraf[2490]: 2021-01-09T00:42:15Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics Jan 09 11:50:55 adelvps telegraf[2490]: 2021-01-09T00:50:55Z E! [inputs.mqtt_consumer] Error in plugin: connection lost: pingresp not received, disconnecting Jan 09 11:50:55 adelvps telegraf[2490]: 2021-01-09T00:50:55Z D! [inputs.mqtt_consumer] Disconnected [tcp://router.au.thethings.network:1883] Jan 09 11:50:55 adelvps telegraf[2490]: 2021-01-09T00:50:55Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics Jan 09 11:51:00 adelvps telegraf[2490]: 2021-01-09T00:51:00Z D! [inputs.mqtt_consumer] Connecting [tcp://router.au.thethings.network:1883] Jan 09 11:51:00 adelvps telegraf[2490]: 2021-01-09T00:51:00Z D! [outputs.influxdb] Wrote batch of 14 metrics in 13.657432ms Jan 09 11:51:00 adelvps telegraf[2490]: 2021-01-09T00:51:00Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics Jan 09 11:51:00 adelvps telegraf[2490]: 2021-01-09T00:51:00Z I! [inputs.mqtt_consumer] Connected [tcp://router.au.thethings.network:1883] Jan 09 11:51:05 adelvps telegraf[2490]: 2021-01-09T00:51:05Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics jan 09 11:51:10 adelvps telegraf[2490]: 2021-01-09T00:51:10Z D! [inputs.mqtt_consumer] Previous collection has not completed; scheduled collection skipped Jan 09 11:51:10 adelvps telegraf[2490]: 2021-01-09T00:51:10Z W! [inputs.mqtt_consumer] Collection took longer than expected; not complete after interval of 10s Jan 09 11:51:10 adelvps telegraf[2490]: 2021-01-09T00:51:10Z D! [outputs.influxdb] Wrote batch of 14 metrics in 11.10536ms Jan 09 11:51:10 adelvps telegraf[2490]: 2021-01-09T00:51:10Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics Jan 09 11:51:15 adelvps telegraf[2490]: 2021-01-09T00:51:15Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics Jan 09 11:51:20 adelvps telegraf[2490]: 2021-01-09T00:51:20Z W! [inputs.mqtt_consumer] Collection took longer than expected; not complete after interval of 10s Jan 09 11:51:20 adelvps telegraf[2490]: 2021-01-09T00:51:20Z D! [inputs.mqtt_consumer] Previous collection has not completed; scheduled collection skipped Jan 09 11:51:20 adelvps telegraf[2490]: 2021-01-09T00:51:20Z D! [outputs.influxdb] Wrote batch of 14 metrics in 9.045908ms Jan 09 11:51:20 adelvps telegraf[2490]: 2021-01-09T00:51:20Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics Jan 09 11:51:25 adelvps telegraf[2490]: 2021-01-09T00:51:25Z D! [outputs.influxdb] Buffer fullness: 0 / 10000 metrics

` I am also running this on a second host running Telegraf, it experienced the same disconnect, and then when it reconnected but no pingrresponse was seen, so disconnects, reconnects, and then fails to get data.

BTW I have two MQTT consumers to the same server. The problem is occurring even if I only have one.