search

influxdata / telegraf

Agent for collecting, processing, aggregating, and writing metrics, logs, and other arbitrary data.
https://influxdata.com/telegraf
MIT License
14.51k stars 5.56k forks source link

JSON_v2 parser error when parsing json array #10646

Open Yahya222 opened 2 years ago

Yahya222 commented 2 years ago

Relevant telegraf.conf

[agent]
  interval = "100s"
  debug = true

[[inputs.file]]
    files = ["20211209.json"]
    data_format = "json_v2"

    [[inputs.file.json_v2]]
        measurement_name = "metric"

        [[inputs.file.json_v2.object]]
            path = "{time:result.0.data.0.timestamps,metric1:result.0.data.0.values,metric2:result.0.data.1.values}"
            tags = ['time','metric1','metric2']
            timestamp_format="unix_ms" 
[[outputs.file]]
  ## Files to write to, "stdout" is a specially handled file.
  files = ["stdout", "metrics4.out"]

Logs from Telegraf

2022-02-14T12:58:41Z I! Starting Telegraf 1.21.3 2022-02-14T12:58:41Z I! Loaded inputs: file 2022-02-14T12:58:41Z I! Loaded aggregators: 2022-02-14T12:58:41Z I! Loaded processors: 2022-02-14T12:58:41Z I! Loaded outputs: file 2022-02-14T12:58:41Z I! Tags enabled: host=BE1CT887 2022-02-14T12:58:41Z I! [agent] Config: Interval:1m40s, Quiet:false, Hostname:"BE1CT887", Flush Interval:10s 2022-02-14T12:58:41Z D! [agent] Initializing plugins 2022-02-14T12:58:41Z D! [agent] Connecting outputs 2022-02-14T12:58:41Z D! [agent] Attempting connection to [outputs.file] 2022-02-14T12:58:41Z D! [agent] Successfully connected to outputs.file 2022-02-14T12:58:41Z D! [agent] Starting service inputs 2022-02-14T12:58:51Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T12:59:01Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T12:59:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T12:59:21Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T12:59:31Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T12:59:41Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T12:59:51Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:00:01Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:00:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:00:21Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:00:31Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:00:41Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:00:51Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:01Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:21Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:31Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:41Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:51Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:02:01Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:02:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:02:21Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:02:31Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:02:41Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:02:51Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:03:01Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:03:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:03:21Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

System info

telegraf-1.21.3

Docker

No response

Steps to reproduce

When I run the query I get this json {"time":[ 1638486300000, 1638486600000, 1638486900000, 1638487200000, 1638487500000, 1638487800000, 1638488100000, 1638488400000, 1638488700000, 1638489000000, 1638489300000, 1638489600000, 1638489900000, 1638490200000, 1638490500000, 1638490800000, 1638491100000, 1638491400000, 1638491700000, 1638492000000, 1638492300000, 1638492600000, .... ],"metric1":[ 2140, 1718, 1532, 2003, 2486, 1845, 1807, 1405, 1870, 1040, 1404, 886, 2599, 1092, 1349, 1088, 1244, 867, 1332, 1328, 1677, 1776, 1782, 1216, 1957, 2432, ... ],"metric2":[ 1306, 1340, 898, 1227, 1707, 1553, 1319, 1060, 1122, 778, 749, 701, 2394, 810, 901, 930, 904, 794, 983, 1235, 942, ... ]} 11Running with default timestamp works but as soon as I try to give time stamp I get this error

Expected behavior

metric metric1=.. metric2=...

Actual behavior

I get nothing just

2022-02-14T13:00:51Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:01Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:11Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:21Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics 2022-02-14T13:01:31Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics

Additional info

I tried different variants including using the timestamp key. I also tried the data converter and starlark.

sspaink commented 2 years ago

@Yahya222 you might be facing this same issue: https://github.com/influxdata/telegraf/issues/10606, selecting the timestamp does seem broken in v1.21.3. There is already a PR that I think resolves it: https://github.com/influxdata/telegraf/pull/10618

Yahya222 commented 2 years ago 
  [[inputs.file]]
      files = ["20211209.json"]
      data_format = "json_v2"

      [[inputs.file.json_v2]]
          measurement_name = "metric"

          [[inputs.file.json_v2.object]]

              path = "{stamp:result.0.data.0.timestamps,metric1:result.0.data.0.values,metric2:result.0.data.1.values}"
              #timestamp_key='stamp'
              #timestamp_format="unix_ms"
              [[inputs.file.json_v2.object.field]]
                path="metric1"
                type= "int"
              [[inputs.file.json_v2.object.field]]
                path="metric2"
                type= "int"
              [[inputs.file.json_v2.object.field]]
                path="stamp"
                type= "int"
              disable_prepend_keys=true

  [[outputs.file]]
    files = ["stdout", "metrics4.out"]

I retried with 1.22 using this input as soon as I remove the comments from timestamp_key it bugs I dont even get any error message. The error is still persistent

Yahya222 commented 2 years ago

I figured out a workaround. I used starlark processor. The issue in my opinion is handling the unix_ms with integer. I suspect it is an error while converting int to string. Therefore when I used the starlark I first converted it to int in the first part and then I used the starlark processor to convert it back to string and assign it to metric.time

kgrimsby commented 2 years ago

Hi @Yahya222 ,

Can you provide code for what you did? I have a similar problem.

Yahya222 commented 2 years ago 
    [[inputs.http.json_v2]]
        measurement_name = "metric3"

        [[inputs.http.json_v2.object]]

            path = "{stamp:result.0.data.0.timestamps,metric1:result.0.data.0.values,metric2:result.0.data.1.values}"

            [[inputs.http.json_v2.object.field]]
              path="metric1"
              type= "int"
            [[inputs.http.json_v2.object.field]]
              path="metric2"
              type= "int"
            [[inputs.http.json_v2.object.field]]
              path="stamp"
              type= "int"
[[processors.starlark]]
 source = '''
load("logging.star", "log")  
def apply(metric):
  log.info(metric)
  metric.time=metric.fields['stamp']*1000000

  return metric
'''
Yahya222 commented 2 years ago

@sspaink @kgrimsby on a complitely unrelated question. Do you have any idea how to change the _start and _stop times when writing a metric into influxdb