Sensu Handler not writing to InfluxDB database

[aumkarpraja]

So essentially the issue I'm having is that the handler is set up correctly according to tutorial provided here: https://blog.sensu.io/check-output-metric-extraction-with-influxdb-grafana

Unfortunately it seems that even on success in a write, InfluxDB sees this and does the write successfully:

Oct 19 15:53:19 ip-172-31-42-92.ec2.internal hab[1194]: influxdb.default(O): [httpd] some.ip - sensu [19/Oct/2018:15:53:19 +0000] "POST /write?consistency=&db=site&precision=s&rp= HTTP/1.1" 204 0 "-" "InfluxDBClient" 19623eed-d3b7-11e8-8006-000000000000 87

But it doesn't reflect accordingly, the database remains empty. I've narrowed it down to a timestamp issue but even after setting the timestamp in the output. The output that the handler gets is as follows.

"output":"cpu,status=warn,cpu_usage=86 1539964880444000000"

The output of sensuctl event info hab-sensu-agent check-cpu --format json.

The checks are located here, I've only currently fully modified check-cpu for now. https://github.com/aumkarpraja/sensu-win-checks/blob/sensu-win-metrics/check_cpu.ps1

output.txt

[aumkarpraja]

As an update to this, I realized my formatting of the output itself was wrong (not matching InfluxDB_line formatting). I've reformatted it but it still doesn't populate the DB. Below is what sensu lists as the output

cpu,name=someinstance status="ok",cpu_usage=22i 1539980719

[nikkictl]

Thanks so much for filing this @aumkarpraja! After looking at your output, it appears that metric extraction did not work. The handler should return an error in this case, I'll verify that happens. Please reference the influx syntax docs (ex. weather,location=us-midwest temperature=82 1465839830100400200) to verify the check's output is in the expected format. There should be error logs on the agent side indicating what exactly is wrong in the metric parsing unable to extract metric from check output.

  "metrics": {
    "handlers": [
      "influxdb"
    ],
    "points": null
  }

[aumkarpraja]

So I did end up formatting it appropriately on Monday, the output was now correct but it ended up still not working with no errors being thrown. I was looking for errors but see Influx receiving the connections just fine.

[nikkictl]

Alright, I understand what's going on now. While your check output cpu,name=someinstance status="ok",cpu_usage=22i 1539980719 is valid influx syntax, only float values are currently supported in Sensu Metric Format. Your agent should be logging the error unable to extract metric from check output and metric field value is invalid, must be a float. That is why you are not seeing metrics being extracted from check output, and that is why your InfluxDB is not being populated. I went ahead and filed https://github.com/sensu/sensu-go/issues/2213 to address the issue in Sensu. For the time being, you could work around the issue by ensuring all values are floats ex: cpu,name=someinstance status=0,cpu_usage=22 1539980719.

[nikkictl]

I'll dig into things on the handler end here, since you mentioned the handler executed with no error {"component":"pipelined","environment":"default","handler":"influxdb","level":"info","msg":"event pipe handler executed","organization":"default","output":"","status":0,"time":"2018-10-19T13:58:19Z"} when it should actually return the error: https://github.com/nikkiki/sensu-influxdb-handler/blob/3f7dc4a19095ead19430e976c5fa95d6ebe69f72/main.go#L99

[nikkictl]

Handler issue has been addressed in https://github.com/nikkiki/sensu-influxdb-handler/commit/5bbc87f448b68c306c35f097626b1e9533d0a526 and will be available in v1.8. Metric extraction issue will be addressed in https://github.com/sensu/sensu-go/issues/2213. Please feel free to re-open if anything else you need was not addressed 😄