influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.6k stars 5.57k forks source link

Unix ms not parsed correctly #10786

Closed tgirotto closed 2 years ago

tgirotto commented 2 years ago

Relevant telegraf.conf

data_format = "json_v2"

  [[inputs.http_listener_v2.json_v2]]
    measurement_name = "customers"

    [[inputs.http_listener_v2.json_v2.object]]
      path = "data"
      timestamp_key = "utc"
      timestamp_format = "unix_ms"

      included_keys = [
        "id",
        "full_name",
      ]

Logs from Telegraf

2022-03-07T15:14:22Z D! [outputs.influxdb_v2] Buffer fullness: 0 / 10000 metrics

System info

Telegraf 1.21.4, Linux 4.15.0-169-generic #177-Ubuntu SMP

Docker

No response

Steps to reproduce

{
    "data" : [
        {
            "id": 449083,
            "full_name": "first customer",
            "grid_id" : 1,
            "utc": 1646666923000,
            "grid_name" : "first customer",
            "grid_type_id" : 1,
            "grid_type_name" : "some grid type",
            "developer_id" : 1,
            "developer_name" : "asd",
            "is_woman" : true,
            "session_id" : 114
        }
    ]
}

Expected behavior

2022-03-07T15:28:43.000000000Z

Actual behavior

2022-03-07T15:28:49.606396290Z

Additional info

No response

powersj commented 2 years ago

Hmm, if you add "utc" to the list of included_keys does it resolve the issue?

customers id=449083,full_name="first customer" 1646666923000000000
[agent]
  omit_hostname = true

[[outputs.file]]

[[inputs.file]]
  files = ["test.json"]
  data_format = "json_v2"
  [[inputs.file.json_v2]]
    measurement_name = "customers"
    [[inputs.file.json_v2.object]]
      path = "data"
      timestamp_key = "utc"
      timestamp_format = "unix_ms"
      included_keys = [
        "id",
        "utc",
        "full_name",
      ]

@sspaink is there a better way to do this?

sspaink commented 2 years ago

I don't think there is a better way to do this, your solution seems like the best approach.

telegraf-tiger[bot] commented 2 years ago

Hello! I am closing this issue due to inactivity. I hope you were able to resolve your problem, if not please try posting this question in our Community Slack or Community Page. Thank you!