influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.87k stars 5.59k forks source link

Records are stagged and seperated with NULLs in the DB when using postgresql as the output #16207

Open zbeucler-largo opened 3 days ago

zbeucler-largo commented 3 days ago

Relevant telegraf.conf

[global_tags]
    project = "test"

[[outputs.postgresql]]
    connection = "host=timescaledb user=myuser password=mypassword sslmode=disable dbname=metrics"
    create_templates = ['''CREATE TABLE {{ .table }} ( {{ .columns }} )''']
    schema = "public"
    timestamp_column_name = "time"

[[processors.date]]
    date_format = "unix_ms"
    timezone = "UTC"
    field_key = "timestamp"

[[inputs.opcua]]
    name = "opcua"
    endpoint = "opc.tcp://my.plc.ip.addr:portnumber"
    connection_timeout = "10s"
    auth_method = "Anonymous"
    security_mode = "None"
    security_policy = "None"
    timestamp = "gather"
    interval = "1s"

    [[inputs.opcua.group]]
        name = "table0"
        nodes = [
            {name="column_a", namespace="2", identifier_type="s", identifier="[device]/path/to/column_a"},
            {name="column_b", namespace="2", identifier_type="s", identifier="[device]/path/to/column_b"},
            {name="column_c", namespace="2", identifier_type="s", identifier="[device]/path/to/column_c"},
        ]

    [[inputs.opcua.group]]
        name = "table1"
        nodes = [
            {name="column_a", namespace="2", identifier_type="s", identifier="[device]/path/to/column_a"},
            {name="column_b", namespace="2", identifier_type="s", identifier="[device]/path/to/column_b"},
            {name="column_c", namespace="2", identifier_type="s", identifier="[device]/path/to/column_c"},
        ]

Logs from Telegraf

2024-11-19 12:01:18 2024-11-19T17:01:18Z I! Loading config: /etc/telegraf/telegraf.conf
2024-11-19 12:01:18 2024-11-19T17:01:18Z I! Starting Telegraf 1.32.0 brought to you by InfluxData the makers of InfluxDB
2024-11-19 12:01:18 2024-11-19T17:01:18Z I! Available plugins: 235 inputs, 9 aggregators, 32 processors, 26 parsers, 62 outputs, 6 secret-stores
2024-11-19 12:01:18 2024-11-19T17:01:18Z I! Loaded inputs: opcua
2024-11-19 12:01:18 2024-11-19T17:01:18Z I! Loaded aggregators:
2024-11-19 12:01:18 2024-11-19T17:01:18Z I! Loaded processors: date
2024-11-19 12:01:18 2024-11-19T17:01:18Z I! Loaded secretstores:
2024-11-19 12:01:18 2024-11-19T17:01:18Z I! Loaded outputs: postgresql
2024-11-19 12:01:18 2024-11-19T17:01:18Z I! Tags enabled: host=telegraf project=test
2024-11-19 12:01:18 2024-11-19T17:01:18Z I! [agent] Config: Interval:10s, Quiet:false, Hostname:"telegraf", Flush Interval:10s
2024-11-19 12:01:18 2024-11-19T17:01:18Z E! [outputs.postgresql] PG connect failed - map[err:failed to connect to `host=timescaledb user=myuser database=metrics`: dial error (dial tcp 172.18.0.2:5432: connect: connection refused)]
2024-11-19 12:01:18 2024-11-19T17:01:18Z E! [agent] Failed to connect to [outputs.postgresql], retrying in 15s, error was "failed to connect to `host=timescaledb user=myuser database=metrics`: dial error (dial tcp 172.18.0.2:5432: connect: connection refused)"
2024-11-19 12:01:34 2024-11-19T17:01:34Z W! [inputs.opcua] Failed to load certificate: open /etc/telegraf/cert.pem: no such file or directory

System info

Telegraf docker image version 1.32.0 (telegraf:1.32.0)

Docker

The host of the container is a air-gapped instance of ubuntu22.04 server

Steps to reproduce

  1. Ensure the PLC hosting the OPCUA server is running
  2. Ensure the TimeseriesDB container is running
  3. Run the telegraf container

Expected behavior

I am expecting the records line up on the same timestamps, and not contain NULLs.

timestamp column_a column_b column_c
1 10 12.5 45
2 11.5 13 46

Actual behavior

The database is filling up with NULL values, and the data appears staggered across rows. This causes problems with queries like: select "column_a" - "column_b" AS "column_d" from table0; Because of the NULLs, column_d ends up being entirely NULL.

timestamp column_a column_b column_c
1 10 null null
1 null 12.5 null
1 null null 45
2 11.5 null null
2 null 13 null
2 null null 46

Additional info

The TimeseriesDB docker image version I am using is timescale/timescaledb:2.16.1-pg16