mqtt_consumer timestamp_path breaks configuration

JaegerStephan commented 2 years ago

Relevant telegraf.conf

[[inputs.mqtt_consumer]]
servers = ["tcp://localhost:1883"]
topics = ["#",]
data_format = "json_v2"
[[inputs.mqtt_consumer.topic_parsing]]
topic = "+/+"
tags = "Line/Measure"
[[inputs.mqtt_consumer.json_v2]]
timestamp_path = "time"
timestamp_format = "unix_ms"
[[inputs.mqtt_consumer.json_v2.field]]
path = "position"
rename = "p"
type = "int"

Logs from Telegraf

nothing particular regarding [[input.mqtt_consumer]]

System info

Telegraf 1.21.4 ,Ubuntu 20.04, docker 20.10.12

Docker

No response

Steps to reproduce

when having the MQTT input plugin with json_v2 data format [[inputs.mqtt_consumer.json_v2]]
whenever timestamp_path and timestamp_format is used all the configuration after the timestamp_path and timestamp_format is ignored and timestamp is also ignored

Expected behavior

Using timestamp_path = "time" and timestamp_format = "unix_ms" should read the timestamp from the MQTT payload and the defined [[inputs.mqtt_consumer.json_v2.field]] should be renamed (from "position" to "p") and appropriately formated (type).

Actual behavior

Whenever the timestamp_path and timestamp_format are used, whether the timestamp is applied correctly to the influx database (always the current time is applied) nor the field is transformed correctly (rename). If I define the timestamp_path and timestamp_format after the [[inputs.mqtt_consumer.json_v2.field]], at least the field is transformed correctly although the timestamp is never applied correctly.

Additional info

MQTT Message

topic: 01/06

payload: { "time": "1646052034000", "position": 55 }

reimda commented 2 years ago

Hi @JaegerStephan, I don't see any problems with the telegraf.conf that you shared, so it looks like this could be a code problem.

It would be easier to debug and fix this if we knew if it happened with the file input. That doesn't require any setup like an mqtt server. Do you know if this bug happens with other input plugins?

Member0815 commented 1 year ago

Hi, I think there are much more problems concerning extracting of timestamps with the json_v2 parser in the telegraf.conf file even in the current telegraf version 1.25:

Sequence of commands timestamp_path, timestamp_format and timestamp_timezone
Problems with timezones != UTC, e.g. "Europe/Berlin"
Working with special timestamp_formats, like in my example at the bottom
Ignoring some or all of the follwing commandlines till next paragraph, resulting in strange error messages starting telegraf service or missing output on [[outputs.file]] in most of the cases an no info whats going wrong in Debug

I've read many, many articles, issues and Readmes and tried roundabout 30 configurations of [[inputs.mqtt_consumer]] with json_v2 parser, without any reasonable result!

I'm using the following setup consisting of

Dietpi (Debian Bullseye) on Raspberry Pi 3B
InfluxDB 1.8.10
Telegraf 1.25
Iobroker with included MQTT Mosquitto Broker
ESP8266 based device sending MQTT-Messages concerning energy consumption like: Topic: /tele/newtek/POWER2 Payload: { "TIME": "2023-01-12T19:08:35", "ENERGY": { "totkWh": 0, "totCnt": 0, "todaykWh": 0, "currW": 0 } }

powersj commented 11 months ago

@JaegerStephan - are you still seeing an issue with the timestamp path? If so, as reimda asked for, could you provide an example config, what behavior you are seeing, and your expected behavior?

Thanks

JaegerStephan commented 11 months ago

Hi @powersj,

unfortunately we are not using it anymore, so I cannot provide you any further information. From my point of view the issue can thus be closed.

Bests and thank you

Member0815 commented 11 months ago

Hi,

I’m still using Telegram, newest version.

In January 2023 I found a solution for me with the processor plugin to get around my problems after trying roundabout 50 different telegraf.conf samples.

They did not work for me using the timestamp path generated and buffered by my microcontroller. Maybe I didn’t understand all details, but some effects shouldn’t have happened like that.

If the current version of Telegram has some improvements or corrections on the timestamp path, we should close the issue waiting for a current example not working properly.

Thanks for all the work

Bernhard

Von: Joshua Powers @.*** Gesendet: Freitag, 17. November 2023 15:21 An: influxdata/telegraf Cc: Member0815; Comment Betreff: Re: [influxdata/telegraf] mqtt_consumer timestamp_path breaks configuration (Issue #10753)

@JaegerStephan https://github.com/JaegerStephan - are you still seeing an issue with the timestamp path? If so, as reimda asked for, could you provide an example config, what behavior you are seeing, and your expected behavior?

Thanks

— Reply to this email directly, view it on GitHub https://github.com/influxdata/telegraf/issues/10753#issuecomment-1816516199 , or unsubscribe https://github.com/notifications/unsubscribe-auth/A3V3KAMBE3VRID75NZ3BZY3YE5XFLAVCNFSM5PTSM7XKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBRGY2TCNRRHE4Q . You are receiving this because you commented. https://github.com/notifications/beacon/A3V3KAIKBMK747KAY5WBDODYE5XFLA5CNFSM5PTSM7XKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGONRC5MZY.gif Message ID: @.***>

powersj commented 11 months ago

Thank you both!

influxdata / telegraf