Input json http configuration misunderstanding

[chi-miel]

Hello,

I am trying to configure a JSON http input, i would like my telegraf to hit my http api.

I made a homemade http api seem like this : { "kafka":{ "kafka1":{ "discarded.full":0, "discarded.nf":0, "enqueued":0, "full":0, "maxqsize":0, "size":0 }, "kafka2":{ "discarded.full":0, "discarded.nf":0, "enqueued":0, "full":0, "maxqsize":10, "size":0 } }, "main":{ "main1":{ "discarded.full":0, "discarded.nf":0, "enqueued":0, "full":0, "maxqsize":0, "size":0 }, "main2":{ "discarded.full":0, "discarded.nf":0, "enqueued":5, "full":0, "maxqsize":12, "size":0 } } }

My input configuration is just: data_format = "json"

And when i test it with : telegraf --config /etc/telegraf/telegraf.d/rsyslog.conf -debug --test I retrieve some data : rsyslog,host=#########################=rsyslog,url=http://=#########################=/ kafka_kafka1_discarded.full=0,kafka_kafka1_discarded.nf=0,kafka_kafka1_enqueued=0,kafka_kafka1_full=0,kafka_kafka1_id=1,kafka_kafka1_maxqsize=0,kafka_kafka1_size=0,kafka_kafka2_discarded.full=0,kafka_kafka2_discarded.nf=0,kafka_kafka2_enqueued=0,kafka_kafka2_full=0,kafka_kafka2_id=2,kafka_kafka2_maxqsize=10,kafka_kafka2_size=0,main_main1_discarded.full=0,main_main1_discarded.nf=0,main_main1_enqueued=0,main_main1_full=0,main_main1_id=1,main_main1_maxqsize=0,main_main1_size=0,main_main2_discarded.full=0,main_main2_discarded.nf=0,main_main2_enqueued=0,main_main2_full=0,main_main2_id=2,main_main2_maxqsize=45,main_main2_size=0 1604425098000000000

But there is nothing in my InfluxDB. (i use chronograf to visualise it)

How can i get data in my influxDB ? I would like retrieve all metrics in this JSON.

Thank you for reading and any help is welcome =)

[srebhan]

@chi-miel how does your output section look like? Is there any error message in the logs?

[chi-miel]

hello @srebhan ,

i not sure to understand because for me output is like i said : rsyslog,host=#########################=rsyslog,url=http://=#########################=/ kafka_kafka1_discarded.full=0,kafka_kafka1_discarded.nf=0,kafka_kafka1_enqueued=0,kafka_kafka1_full=0,kafka_kafka1_id=1,kafka_kafka1_maxqsize=0,kafka_kafka1_size=0,kafka_kafka2_discarded.full=0,kafka_kafka2_discarded.nf=0,kafka_kafka2_enqueued=0,kafka_kafka2_full=0,kafka_kafka2_id=2,kafka_kafka2_maxqsize=10,kafka_kafka2_size=0,main_main1_discarded.full=0,main_main1_discarded.nf=0,main_main1_enqueued=0,main_main1_full=0,main_main1_id=1,main_main1_maxqsize=0,main_main1_size=0,main_main2_discarded.full=0,main_main2_discarded.nf=0,main_main2_enqueued=0,main_main2_full=0,main_main2_id=2,main_main2_maxqsize=45,main_main2_size=0 1604425098000000000

If this is not ouput, can you tell me which command to test it ?

There no error log because in telegraf.

Thank you

[srebhan]

Hey, in your config file there should be an [[outputs.influxdb]] or [[outputs.influxdb_v2]] section, specifying where to write the data to... Otherwise telegraf cannot know where it should send the data to. :-)

[chi-miel]

Hello,

yes in my telegraf.conf, there is :

[[outputs.influxdb]]
  database = "rsyslog"

[srebhan]

So there is no URL given in your output which means it will try the default one (http://localhost:8086). Is this where your influxdb server is running?

Ore there any errors/warnings in the log-file or console output?

[chi-miel]

Hey,

Yes my influxdb listen in 8086, i knwo it's working because i have other telegraf input which insert data into my infludb.

In telegraf log, i have some error from other plugin (vmware).

Whe i'm filtering with my url homemade api, there no log (warning and error).

I test an new json structure like : [{"type":"kafka","queue":"main","data":{"size":0,"enqueued":0,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":0}},{"type":"kafka","queue":"DA","data":{"size":0,"enqueued":0,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":0}},{"type":"main","queue":"main","data":{"size":0,"enqueued":0,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":0}},{"type":"main","queue":"DA","data":{"size":0,"enqueued":0,"full":0,"discarded.full":0,"discarded.nf":0,"maxqsize":0}}]

and i have configure my input like this : data_format = "json" tag_keys = ["type", "queue"]

It's work, i see data in my influxdb but i don't understand why my first json not working.

My first json have bad structure ?

[srebhan]

I'm pretty sure it's not your input, i.e. it should not be the json that causes the problem. The problem is most likely that you specified the --test flag. According to the documentation of the flag "[...] processors, aggregators and outputs are not run"!

[srebhan]

@chi-miel did that fix your issue? If yes, please close that issue here. :-)

[chi-miel]

Hello,

Sorry for the delay, i test my input by restarting telegraf, but no data in influxdb.

I will investigate more deeply tommrow

Thank you

[srebhan]

This is really strange... So you restart the telegraf without specifying the --test flag, see the output on console, but not in the database? If so, could you add your whole setup here for me to reproduce the issue!?

[chi-miel]

Hello @srebhan ,

Thank you, for your help ! =)

I use other plugin, don't bother about them : telegraf.conf

[[outputs.influxdb]]
  database = "vmware"
  [outputs.influxdb.tagdrop]
    tag1 = [ "graylog", "elasticsearch", "kafka", "zookeeper", "rsyslog" ]

[[outputs.influxdb]]
  database = "graylog"
  [outputs.influxdb.tagpass]
    tag1 = [ "graylog" ]

[[outputs.influxdb]]
  database = "elasticsearch"
  [outputs.influxdb.tagpass]
    tag1 = [ "elasticsearch" ]

[[outputs.influxdb]]
  database = "kafka"
  [outputs.influxdb.tagpass]
    tag1 = [ "kafka" ]

[[outputs.influxdb]]
  database = "zookeeper"
  [outputs.influxdb.tagpass]
    tag1 = [ "zookeeper" ]

[[outputs.influxdb]]
  database = "rsyslog"
  [outputs.influxdb.tagpass]
    tag1 = [ "rsyslog" ]

and telegraf.d/rsyslog.conf

[[inputs.http]]
  name_override = "rsyslog"
  urls = ["http://####################.com:5000/"]
  headers = {"Content-Type" = "application/json"}
  data_format = "json"
  [inputs.http.tags]
    tag1 = "rsyslog"

[sjwang90]

if you add the outputs.file with -test do you see data in your terminal?

[[outputs.file]]
files = ["stdout"]

[chi-miel]

hello, sorry, i didn't have time to test it today, i wil try next week, thank you!

[chi-miel]

Hello @sjwang90 ,

i test it with my tag : telegraf.d/rsyslog.conf

[[outputs.file]]
  files = ["stdout"]
  tag1 = [ "rsyslog" ]

then i restarted telegraf and run test command :

telegraf --config /etc/telegraf/telegraf.d/rsyslog.conf -debug --test

on my terminal , i see :

2020-11-10T09:08:37Z I! Starting Telegraf 1.16.1
2020-11-10T09:08:37Z D! [agent] Initializing plugins
2020-11-10T09:08:37Z D! [agent] Starting service inputs
2020-11-10T09:08:37Z D! [agent] Stopping service inputs
2020-11-10T09:08:37Z D! [agent] Input channel closed
2020-11-10T09:08:37Z D! [agent] Stopped Successfully
> rsyslog,host=######################,tag1=rsyslog,url=http://######################:5000/ kafka_kafka1_discarded.full=0,kafka_kafka1_discarded.nf=0,kafka_kafka1_enqueued=0,kafka_kafka1_full=0,kafka_kafka1_id=1,kafka_kafka1_maxqsize=0,kafka_kafka1_size=0,kafka_kafka2_discarded.full=0,kafka_kafka2_discarded.nf=0,kafka_kafka2_enqueued=0,kafka_kafka2_full=0,kafka_kafka2_id=2,kafka_kafka2_maxqsize=10,kafka_kafka2_size=0,main_main1_discarded.full=0,main_main1_discarded.nf=0,main_main1_enqueued=0,main_main1_full=0,main_main1_id=1,main_main1_maxqsize=0,main_main1_size=0,main_main2_discarded.full=0,main_main2_discarded.nf=0,main_main2_enqueued=0,main_main2_full=0,main_main2_id=2,main_main2_maxqsize=45,main_main2_size=0 160499931800000000

[srebhan]

@chi-miel did you test with influx outputh without --test in the meantime?

[chi-miel]

Hello,

without --test, it not work :

2020-11-10T09:29:37Z E! [telegraf] Error running agent: Error: no outputs found, did you provide a valid config file?

i think because with this command telegraf doesn't the whole configuration , telegraf configuration is /etc/telegraf/telegraf.conf and when i use --test , i specify only my input rsyslog configuration /etc/telegraf/telegraf.d/rsyslog.conf

[sjwang90]

can you share your whole /etc/telegraf/telegraf.d/rsyslog.conf config?

[chi-miel]

hello,

yes , I use other plugin, don't bother about them : telegraf.conf

[[outputs.influxdb]]
  database = "vmware"
  [outputs.influxdb.tagdrop]
    tag1 = [ "graylog", "elasticsearch", "kafka", "zookeeper", "rsyslog" ]

[[outputs.influxdb]]
  database = "graylog"
  [outputs.influxdb.tagpass]
    tag1 = [ "graylog" ]

[[outputs.influxdb]]
  database = "elasticsearch"
  [outputs.influxdb.tagpass]
    tag1 = [ "elasticsearch" ]

[[outputs.influxdb]]
  database = "kafka"
  [outputs.influxdb.tagpass]
    tag1 = [ "kafka" ]

[[outputs.influxdb]]
  database = "zookeeper"
  [outputs.influxdb.tagpass]
    tag1 = [ "zookeeper" ]

[[outputs.influxdb]]
  database = "rsyslog"
  [outputs.influxdb.tagpass]
    tag1 = [ "rsyslog" ]

[srebhan]

@chi-miel: If I understand your setup correctly you have a general telegraf.conf and another one for the rsyslog in /etc/telegraf/telegraf.d/rsyslog.conf. If this is correct, you need to load both when starting telegraf by e.g. telegraf --config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d. Alternatively you can merge two file, but that's more for testing.

[chi-miel]

Hello @srebhan ,

I merged two file my telegraf.conf and my rsyslog.conf then i run telegraf --config /etc/telegraf/telegraf.d/test.conf -debug --test

One error on line :

[[outputs.file]]
  files = ["stdout"]
  tag1 = [ "rsyslog" ]

telegraf doesn't know "tag1" some i juste delete tag1 = [ "rsyslog" ] and then run telegraf --config /etc/telegraf/telegraf.d/test.conf -debug --test

output (i filtered with grep ):

telegraf --config /etc/telegraf/telegraf.d/test.conf -debug --test | grep rsyslog | grep pirv-siem-relay-01
2020-11-12T10:28:42Z I! Starting Telegraf 1.16.1
> rsyslog,host=############################,tag1=rsyslog,url=http://############################:5000/ kafka_kafka1_discarded.full=0,kafka_kafka1_discarded.nf=0,kafka_kafka1_enqueued=0,kafka_kafka1_full=0,kafka_kafka1_id=1,kafka_kafka1_maxqsize=0,kafka_kafka1_size=0,kafka_kafka2_discarded.full=0,kafka_kafka2_discarded.nf=0,kafka_kafka2_enqueued=0,kafka_kafka2_full=0,kafka_kafka2_id=2,kafka_kafka2_maxqsize=10,kafka_kafka2_size=0,main_main1_discarded.full=0,main_main1_discarded.nf=0,main_main1_enqueued=0,main_main1_full=0,main_main1_id=1,main_main1_maxqsize=0,main_main1_size=0,main_main2_discarded.full=0,main_main2_discarded.nf=0,main_main2_enqueued=0,main_main2_full=0,main_main2_id=2,main_main2_maxqsize=45,main_main2_size=0 1605176935000000000

[srebhan]

So it seem like you are getting data. What about influxdb? Remember you should not specify --test when testing with influxdb! Just add your [[outputs.influxdb]] section to test.conf and run telegraf --config /etc/telegraf/telegraf.d/test.conf -debug.

[chi-miel]

Hello @srebhan ,

yes it's work ! i have some data into my influxdb database.

I don't understand why it working now, in any case thank you support !

I will keep some tips you give me =) !

Thank you

[srebhan]

When you split your config files, you need to tell telegraf to load them all! I'm usually putting the agent configuration and general setting to /etc/telegraf/telegraf.conf and keep the outputs etc in individual files in /etc/telegraf/telegraf.d (e.g. /etc/telegraf/telegraf.d/outputs_influxdb.conf, /etc/telegraf/telegraf.d/inputs_sensors_A.conf ... /etc/telegraf/telegraf.d/inputs_sensors_Z.conf. With this setup, telegraf needs to be started with telegraf --config /etc/telegraf/telegraf.conf --config-directory /etc/telegraf/telegraf.d.

It's important to note that the config-files need to end with .conf in the specified config-directory as otherwise they are not found and not loaded by telegraf!