influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.63k stars 5.58k forks source link

mqtt_consumer timestamp_path breaks configuration #10753

Closed JaegerStephan closed 11 months ago

JaegerStephan commented 2 years ago

Relevant telegraf.conf

[[inputs.mqtt_consumer]]
servers = ["tcp://localhost:1883"]
topics = ["#",]
data_format = "json_v2"
[[inputs.mqtt_consumer.topic_parsing]]
topic = "+/+"
tags = "Line/Measure"
[[inputs.mqtt_consumer.json_v2]]
timestamp_path = "time"
timestamp_format = "unix_ms"
[[inputs.mqtt_consumer.json_v2.field]]
path = "position"
rename = "p"
type = "int"

Logs from Telegraf

nothing particular regarding [[input.mqtt_consumer]]

System info

Telegraf 1.21.4 ,Ubuntu 20.04, docker 20.10.12

Docker

No response

Steps to reproduce

  1. when having the MQTT input plugin with json_v2 data format [[inputs.mqtt_consumer.json_v2]]
  2. whenever timestamp_path and timestamp_format is used all the configuration after the timestamp_path and timestamp_format is ignored and timestamp is also ignored

Expected behavior

Using timestamp_path = "time" and timestamp_format = "unix_ms" should read the timestamp from the MQTT payload and the defined [[inputs.mqtt_consumer.json_v2.field]] should be renamed (from "position" to "p") and appropriately formated (type).

Actual behavior

Whenever the timestamp_path and timestamp_format are used, whether the timestamp is applied correctly to the influx database (always the current time is applied) nor the field is transformed correctly (rename). If I define the timestamp_path and timestamp_format after the [[inputs.mqtt_consumer.json_v2.field]], at least the field is transformed correctly although the timestamp is never applied correctly.

Additional info

MQTT Message

topic: 01/06

payload: { "time": "1646052034000", "position": 55 }

reimda commented 2 years ago

Hi @JaegerStephan, I don't see any problems with the telegraf.conf that you shared, so it looks like this could be a code problem.

It would be easier to debug and fix this if we knew if it happened with the file input. That doesn't require any setup like an mqtt server. Do you know if this bug happens with other input plugins?

Member0815 commented 1 year ago

Hi, I think there are much more problems concerning extracting of timestamps with the json_v2 parser in the telegraf.conf file even in the current telegraf version 1.25:

I've read many, many articles, issues and Readmes and tried roundabout 30 configurations of [[inputs.mqtt_consumer]] with json_v2 parser, without any reasonable result!

I'm using the following setup consisting of

powersj commented 11 months ago

@JaegerStephan - are you still seeing an issue with the timestamp path? If so, as reimda asked for, could you provide an example config, what behavior you are seeing, and your expected behavior?

Thanks

JaegerStephan commented 11 months ago

Hi @powersj,

unfortunately we are not using it anymore, so I cannot provide you any further information. From my point of view the issue can thus be closed.

Bests and thank you

Member0815 commented 11 months ago

Hi,

I’m still using Telegram, newest version.

In January 2023 I found a solution for me with the processor plugin to get around my problems after trying roundabout 50 different telegraf.conf samples.

They did not work for me using the timestamp path generated and buffered by my microcontroller. Maybe I didn’t understand all details, but some effects shouldn’t have happened like that.

If the current version of Telegram has some improvements or corrections on the timestamp path, we should close the issue waiting for a current example not working properly.

Thanks for all the work

Bernhard

Von: Joshua Powers @.*** Gesendet: Freitag, 17. November 2023 15:21 An: influxdata/telegraf Cc: Member0815; Comment Betreff: Re: [influxdata/telegraf] mqtt_consumer timestamp_path breaks configuration (Issue #10753)

@JaegerStephan https://github.com/JaegerStephan - are you still seeing an issue with the timestamp path? If so, as reimda asked for, could you provide an example config, what behavior you are seeing, and your expected behavior?

Thanks

— Reply to this email directly, view it on GitHub https://github.com/influxdata/telegraf/issues/10753#issuecomment-1816516199 , or unsubscribe https://github.com/notifications/unsubscribe-auth/A3V3KAMBE3VRID75NZ3BZY3YE5XFLAVCNFSM5PTSM7XKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCOBRGY2TCNRRHE4Q . You are receiving this because you commented. https://github.com/notifications/beacon/A3V3KAIKBMK747KAY5WBDODYE5XFLA5CNFSM5PTSM7XKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGONRC5MZY.gif Message ID: @.***>

powersj commented 11 months ago

Thank you both!