Closed icinga-migration closed 8 years ago
Updated by gbeutner on 2016-07-26 05:54:08 +00:00
@spjmurray: Can you have a look at this please? :)
Updated by spjmurray on 2016-07-26 06:00:09 +00:00
I can try escaping the backslashes... Utterly undocumented though by influxdata sigh
Updated by spjmurray on 2016-07-27 14:09:41 +00:00
Recreate...
#!/usr/bin/env python
print "ok | C:\=10;80;90;0;100"
object CheckCommand "windows" {
import "plugin-check-command"
command = [ "/home/simon/check_windows" ]
}
apply Service "windows" {
import "generic-service"
check_command = "windows"
zone = host.name
assign where host.vars.role == "icinga_master"
}
[2016-07-27 15:06:06 +0100] debug/InfluxdbWriter: Add to metric list:'windows,domain=angel.net,fqdn=icinga.angel.net,hostname=icinga,service=windows,metric=C:\ acknowledgement=0i,crit=90,current_attempt=1i,downtime_depth=0i,execution_time=0.022955,latency=0.001167,max=100,max_check_attempts=5i,reachable=true,state=0i,state_type=1i,value=10,warn=80 1469628366'.
[2016-07-27 15:05:11 +0100] warning/InfluxdbWriter: Unexpected response code 400
Updated by spjmurray on 2016-07-27 15:09:48 +00:00
Well this is horrible...
Influx interprets "\ " as an escaped space, likewise "\," as an escaped comma, thus screwing things up in this instance. Escaped backslashes aren't a thing either.
Our only option is to replace trailing slashes (which spoils the prettiness of Grafana), or translate to "\\ " which will confusingly add a magical white space on the end, but look pretty! I prefer the latter as "C:\ " looks more sane to the spreadsheet crowd than "C:_"
Updated by TheFlyingCorpse on 2016-07-27 17:24:27 +00:00
https://docs.influxdata.com/influxdb/v0.13/write\_protocols/write\_syntax/#escaping-characters
are we sure its not icinga that is doing this? I dont see this issue with other sources and have the \ represented (that I can see) without a space at the end.
C:_ wouldnt really be an issue, it would be safer to use when querying than \'s, no?
Updated by TheFlyingCorpse on 2016-07-27 17:26:14 +00:00
And I didnt even read the whole thing before clicking submit.
Backslashes do not need to be escaped when used in strings. Unless followed by a comma, space, or equals sign backslashes are treated as a normal character. Escaping Field Key
What if the string is sent as "C:\" instead? Ie, encapsulate in ""'s if it ends with a \?
Updated by TheFlyingCorpse on 2016-07-27 18:13:54 +00:00
Bah. i see the issue now.
Tried both ,"metric=C:\" and ,metric="C:\", it doesnt look too good either. The issue I have with using "C:\ " is that if you dont use the gui to create it, querying for this will be ugly. If we changed characters as graphite does (\ to _ amongst other characters), its a known way its done and it is a way that is relatively standardized.
Yes, C:_ looks bad but at least its safe and we are then also relatively safe from some future changes that might happen in InfluxDB.
Updated by spjmurray on 2016-07-28 08:27:11 +00:00
Given it's only windows that suffers I'm game O:)
Updated by spjmurray on 2016-07-28 09:03:14 +00:00
Although undocumented it appears to escape '=' too...
> insert windows,tag=C:\ C:\=10 1469630161
ERR: {"error":"unable to parse 'windows,tag=C:\\ C:\\=10 1469630161': invalid field format"}
> insert windows,tag=C:_ C:\=10 1469630161
ERR: {"error":"unable to parse 'windows,tag=C:_ C:\\=10 1469630161': invalid field format"}
> insert windows,tag=C:_ C:_=10 1469630161
>
Updated by spjmurray on 2016-07-28 09:32:23 +00:00
Much better!
[2016-07-28 10:17:22 +0100] debug/InfluxdbWriter: Add to metric list:'windows,domain=angel.net,fqdn=icinga.angel.net,hostname=icinga,service=windows,metric=C:_ acknowledgement=0i,crit=90,current_attempt=1i,downtime_depth=0i,execution_time=0.029854,latency=0.000232,max=100,max_check_attempts=5i,reachable=true,state=0i,state_type=1i,value=10,warn=80 1469697442'.
Updated by spjmurray on 2016-07-28 09:33:58 +00:00
From e605e81f2f9de9e6650d89ba56bae626cf7e640d Mon Sep 17 00:00:00 2001 From: Simon Murray <spjmurray@yahoo.co.uk> Date: Thu, 28 Jul 2016 10:28:53 +0100 Subject: [PATCH] Fix InfluxdbWriter Trailing Backslash
Backslashes escape spaces or commas (and evidently equals), given tags are separated by commas, tag keys and values are separated by equals and tags are separated from fields by a space we need to take action when these end in a backslash e.g. 'C:\'. Also discovered a bug whereby the metric tag was missing out on escaping.
refs #12227
Updated by gbeutner on 2016-07-28 11:41:19 +00:00
The patch file you've attached to this ticket seems to be for another ticket. :)
Updated by spjmurray on 2016-07-28 11:42:53 +00:00
Right patch :)
Updated by spjmurray on 2016-07-28 12:15:05 +00:00
Applied in changeset d1dbe2a02f9d1b6d43e7f2c610dafe223f7b7f71.
Updated by mfriedrich on 2016-07-28 12:22:02 +00:00
Updated by spjmurray on 2016-07-28 12:55:15 +00:00
Less modern patch which will work without -std=c++11 and string::back()
Updated by gbeutner on 2016-07-28 13:55:01 +00:00
Updated by TheFlyingCorpse on 2016-07-30 11:38:35 +00:00
The patch does not fix everything as expected. It adds slashes for escaping what should just be quoted.
Example:
[2016-07-30 13:23:11 +0200] debug/InfluxdbWriter: Add to metric list:'service-windows,hostname=azmap15.labdomain.net,service=WSvc\ RpcSs,metric=service acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max=7,max_check_attempts=3i,min=1,reachable=true,state=0i,state_type=1i,value=4 1469877790'.
It appears this is escaped again after the debug output, as it is being sent to InfluxDB. When I tried to debug this using curl I got the result I wanted and expected, without the \ in the name! I could only reproduce this by adding another \ to service=WSvc\ RpcSs, if I only had one it was in the expected output and not with the extra \.
Steps to reproduce: Wrong:
icinga2@ryglov09:~$ curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'service-windows,hostname=azmap15.labdomain.net,service=WSvc\\ RpcSsB,metric=service acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max=7,max_check_attempts=3i,min=1,reachable=true,state=0i,state_type=1i,value=4 1469752253000000000'
HTTP/1.1 204 No Content
Request-Id: 09be3afe-564a-11e6-ad5d-000000000000
X-Influxdb-Version: 1.0.0-beta3
Date: Sat, 30 Jul 2016 11:37:47 GMT
icinga2@ryglov09:~$ curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'service-windows,hostname=azmap15.labdomain.net,service=WSvc\\ RpcSsB,metric=service acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max=7,max_check_attempts=3i,min=1,reachable=true,state=0i,state_type=1i,value=4 1469952253000000000'
HTTP/1.1 204 No Content
Request-Id: 0d07877d-564a-11e6-ad67-000000000000
X-Influxdb-Version: 1.0.0-beta3
Date: Sat, 30 Jul 2016 11:37:52 GMT
Correct:
icinga2@ryglov09:~$ curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'service-windows,hostname=azmap15.labdomain.net,service=WSvc\ RpcSsA,metric=service acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max=7,max_check_attempts=3i,min=1,reachable=true,state=0i,state_type=1i,value=4 1469742253000000000'
HTTP/1.1 204 No Content
Request-Id: aa82c129-5649-11e6-acb5-000000000000
X-Influxdb-Version: 1.0.0-beta3
Date: Sat, 30 Jul 2016 11:35:07 GMT
icinga2@ryglov09:~$ curl -i -XPOST 'http://localhost:8086/write?db=mydb' --data-binary 'service-windows,hostname=azmap15.labdomain.net,service=WSvc\ RpcSsA,metric=service acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max=7,max_check_attempts=3i,min=1,reachable=true,state=0i,state_type=1i,value=4 1469752253000000000'
HTTP/1.1 204 No Content
Request-Id: ad9810c1-5649-11e6-acb7-000000000000
X-Influxdb-Version: 1.0.0-beta3
Date: Sat, 30 Jul 2016 11:35:12 GMT
Updated by spjmurray on 2016-08-01 11:46:12 +00:00
It isn't escaped again, what gets echoed out is what gets added to a list, then joined with \n into the message body, no more, no less. You should be able to see this via 'tcpdump -i blah0 port 8086 -A'. InfluxDB however doesn't necessarily reply with what you'd expect, and that has been known to display extra slashes. I'd double check what's actually on the wire.
Updated by TheFlyingCorpse on 2016-08-04 02:09:13 +00:00
ServiceName in Icinga2: "SuperDIsk hhh" Capture over the wire:
disk,hostname=azmap17.labdomain.net,service=SuperDIsk\\ hhh,metric=D:_ acknowledgement=0i,crit=751514419.200000,current_attempt=1i,downtime_depth=0i,execution_time=0.016000,latency=0,max=7515144192,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=6807355392,warn=1503028838.400000 147
It does look to me like it is double escaped.
Updated by spjmurray on 2016-08-08 08:53:55 +00:00
[2016-08-08 09:51:21 +0100] debug/InfluxdbWriter: Add to metric list:'test,domain=angel.net,fqdn=icinga.angel.net,hostname=icinga,service=test\ with\ space,metric=C:_ acknowledgement=0i,crit=90,current_attempt=1i,downtime_depth=0i,execution_time=0.021648,latency=0.000454,max=100,max_check_attempts=5i,reachable=true,state=0i,state_type=1i,value=10,warn=80 1470646281'.
[2016-08-08 09:51:21 +0100] notice/ApiListener: Relaying 'event::SetNextCheck' message
[2016-08-08 09:51:21 +0100] notice/ApiListener: Relaying 'event::CheckResult' message
[2016-08-08 09:51:22 +0100] debug/InfluxdbWriter: Timer expired writing 19 data points
[2016-08-08 09:51:22 +0100] notice/InfluxdbWriter: Reconnecting to InfluxDB on host 'influxdb.angel.net' port '8086'.
[2016-08-08 09:51:22 +0100] debug/HttpRequest: line: HTTP/1.1 204 No Content, tokens: 4
Is what I'm getting out which works perfectly, unable to recreate :*(
Updated by mfriedrich on 2016-08-12 08:41:50 +00:00
Is there anything else what needs to be done for resolving/closing this issue?
Updated by TheFlyingCorpse on 2016-08-15 16:40:02 +00:00
ServiceConfig:
template Service "_Icinga2LocalCheck" {
check_interval = 1m
retry_interval = 1m
command_endpoint = host_name
}
template Service "disk-linux" {
import "_Icinga2LocalCheck"
check_command = "disk"
}
template Service "disk-windows" {
import "_Icinga2LocalCheck"
check_command = "disk-windows"
}
template Service "influxDB-disk-linux" {
import "disk-linux"
}
template Service "influxDB-disk-windows" {
import "disk-windows"
}
apply Service "influxDB-Test1" {
import "influxDB-disk-linux"
assign where host.vars.osfamily == "Debian"
}
apply Service "influxDB-Test 2" {
import "influxDB-disk-linux"
assign where host.vars.osfamily == "Debian"
}
apply Service "influxDB-Test3 " {
import "influxDB-disk-linux"
assign where host.vars.osfamily == "Debian"
}
apply Service "influxDB-Test 4 " {
import "influxDB-disk-linux"
assign where host.vars.osfamily == "Debian"
}
apply Service "influxDB-Test1" {
import "influxDB-disk-windows"
assign where host.vars.osfamily == "windows"
}
apply Service "influxDB-Test 2" {
import "influxDB-disk-windows"
assign where host.vars.osfamily == "windows"
}
apply Service "influxDB-Test3 " {
import "influxDB-disk-windows"
assign where host.vars.osfamily == "windows"
}
apply Service "influxDB-Test 4 " {
import "influxDB-disk-windows"
assign where host.vars.osfamily == "windows"
}
Output got via tcpdump, grep filtered to only get the tests:
disk,hostname=ryglif11.labdomain.net,service=influxDB-Test1,metric=/boot/efi acknowledgement=0i,crit=481296384,current_attempt=1i,downtime_depth=0i,execution_time=0.001008,latency=0,max=534773760,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=0,warn=427819008 1471278799
disk,hostname=ryglif11.labdomain.net,service=influxDB-Test\\ 4\\ ,metric=/ acknowledgement=0i,crit=45780828160,current_attempt=1i,downtime_depth=0i,execution_time=0.292895,latency=0,max=50868518912,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=4478468096,warn=40694185984 1471278799
disk,hostname=ryglif11.labdomain.net,service=influxDB-Test\\ 4\\ ,metric=/boot acknowledgement=0i,crit=222298112,current_attempt=1i,downtime_depth=0i,execution_time=0.292895,latency=0,max=247463936,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=34603008,warn=197132288 1471278799
disk,hostname=ryglif11.labdomain.net,service=influxDB-Test\\ 4\\ ,metric=/boot/efi acknowledgement=0i,crit=481296384,current_attempt=1i,downtime_depth=0i,execution_time=0.292895,latency=0,max=534773760,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=0,warn=427819008 1471278799
disk-windows,hostname=azmap17.labdomain.net,service=influxDB-Test\\ 2,metric=C:_ acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max=136363114496,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=123417395200 1471278798
disk-windows,hostname=azmap17.labdomain.net,service=influxDB-Test\\ 2,metric=D:_ acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max=7515144192,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=6807355392 1471278798
disk,hostname=ryglov09.labdomain.net,service=influxDB-Test\\ 4\\ ,metric=/ acknowledgement=0i,crit=45245005824,current_attempt=1i,downtime_depth=0i,execution_time=0.000799,latency=0.000475,max=50272927744,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=7597981696,warn=40218132480 1471278799
disk,hostname=ryglov09.labdomain.net,service=influxDB-Test\\ 4\\ ,metric=/boot acknowledgement=0i,crit=221249536,current_attempt=1i,downtime_depth=0i,execution_time=0.000799,latency=0.000475,max=246415360,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=33554432,warn=197132288 1471278799
disk-windows,hostname=azmap15.labdomain.net,service=influxDB-Test1,metric=C:_ acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max=136363114496,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=122009157632 1471278799
disk-windows,hostname=azmap15.labdomain.net,service=influxDB-Test1,metric=D:_ acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0,latency=0,max=7515144192,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=6121586688 1471278799
disk-windows,hostname=azmap17.labdomain.net,service=influxDB-Test3\\ ,metric=C:_ acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0.016000,latency=0,max=136363114496,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=123417395200 1471278798
disk-windows,hostname=azmap17.labdomain.net,service=influxDB-Test3\\ ,metric=D:_ acknowledgement=0i,current_attempt=1i,downtime_depth=0i,execution_time=0.016000,latency=0,max=7515144192,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=6807355392 1471278798
disk,hostname=ryglif11.labdomain.net,service=influxDB-Test\\ 2,metric=/ acknowledgement=0i,crit=45780828160,current_attempt=1i,downtime_depth=0i,execution_time=0.001725,latency=0,max=50868518912,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=4478468096,warn=40694185984 1471278800
disk,hostname=ryglif11.labdomain.net,service=influxDB-Test\\ 2,metric=/boot acknowledgement=0i,crit=222298112,current_attempt=1i,downtime_depth=0i,execution_time=0.001725,latency=0,max=247463936,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=34603008,warn=197132288 1471278800
disk,hostname=ryglif11.labdomain.net,service=influxDB-Test\\ 2,metric=/boot/efi acknowledgement=0i,crit=481296384,current_attempt=1i,downtime_depth=0i,execution_time=0.001725,latency=0,max=534773760,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=0,warn=427819008 1471278800
disk,hostname=ryglif10.labdomain.net,service=influxDB-Test1,metric=/ acknowledgement=0i,crit=45245005824,current_attempt=1i,downtime_depth=0i,execution_time=0.000981,latency=0,max=50272927744,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=19604176896,warn=40218132480 1471278801
disk,hostname=ryglif10.labdomain.net,service=influxDB-Test1,metric=/boot acknowledgement=0i,crit=221249536,current_attempt=1i,downtime_depth=0i,execution_time=0.000981,latency=0,max=246415360,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=33554432,warn=197132288 1471278801
disk,hostname=ryglov39.labdomain.net,service=influxDB-Test3\\ ,metric=/ acknowledgement=0i,crit=17861443584,current_attempt=1i,downtime_depth=0i,execution_time=0.003766,latency=0,max=19846397952,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=2366636032,warn=15876489216 1471278802
disk,hostname=ryglov39.labdomain.net,service=influxDB-Test3\\ ,metric=/boot acknowledgement=0i,crit=221249536,current_attempt=1i,downtime_depth=0i,execution_time=0.003766,latency=0,max=246415360,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=33554432,warn=197132288 1471278802
disk,hostname=ryglif10.labdomain.net,service=influxDB-Test\\ 4\\ ,metric=/ acknowledgement=0i,crit=45245005824,current_attempt=1i,downtime_depth=0i,execution_time=0.001733,latency=0,max=50272927744,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=19604176896,warn=40218132480 1471278802
disk,hostname=ryglif10.labdomain.net,service=influxDB-Test\\ 4\\ ,metric=/boot acknowledgement=0i,crit=221249536,current_attempt=1i,downtime_depth=0i,execution_time=0.001733,latency=0,max=246415360,max_check_attempts=3i,reachable=true,state=0i,state_type=1i,value=33554432,warn=197132288 1471278802
Icinga2 was compiled from last nights master with ido magic by shroud, the output is similar when using snapshots.
root@ryglov09:~# icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.4.10-1)
Copyright (c) 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Application information:
Installation root: /usr
Sysconf directory: /etc
Run directory: /run
Local state directory: /var
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid
System information:
Platform: Debian GNU/Linux
Platform version: 8 (jessie)
Kernel: Linux
Kernel version: 3.16.0-4-amd64
Architecture: x86_64
Build information:
Compiler: GNU 4.9.2
Build host: deb-i2-host
I'll dig around a bit and see if I can figure out why it double escapes.
Updated by TheFlyingCorpse on 2016-08-16 20:28:12 +00:00
Patches to fix the escaping attached.
Updated by TheFlyingCorpse on 2016-08-16 20:56:16 +00:00
Applied in changeset 93dc0bdaea934c707c59b2e8c5b973e2ceaa1dc2.
Updated by TheFlyingCorpse on 2016-08-16 21:57:05 +00:00
Patch needs to be merged before this is resolved. I goofed.
Updated by TheFlyingCorpse on 2016-08-17 04:10:08 +00:00
Applied in changeset 4e3da7e3aba59f1ba1aee96c45453141e2e51129.
This issue has been migrated from Redmine: https://dev.icinga.com/issues/12227
Created by TheFlyingCorpse on 2016-07-25 18:09:06 +00:00
Assignee: (none) Status: Resolved (closed on 2016-08-17 04:10:08 +00:00) Target Version: 2.5.0 Last Update: 2016-08-17 04:10:08 +00:00 (in Redmine)
Perfdata presented to InfluxdbWriter:
This is the data in icinga2:
This data does not appear in influxdb. Other metrics from the host appear, but not the above.
Attachments
Changesets
2016-07-28 12:11:23 +00:00 by spjmurray d1dbe2a02f9d1b6d43e7f2c610dafe223f7b7f71
2016-07-28 13:54:33 +00:00 by spjmurray 344248f384b5fdb248a32e4dbd0e9e138ce40c94
2016-07-29 04:51:33 +00:00 by spjmurray 84ea0065b2e8f8ffd870ea6915fa6dea31101361
2016-08-16 20:36:09 +00:00 by TheFlyingCorpse bfed90c3afe3d656eb48fa9eca7916ad06974087
2016-08-16 20:54:15 +00:00 by TheFlyingCorpse 35eb65429703fb0a2741dae379fd2761ddde0765
2016-08-16 20:54:24 +00:00 by TheFlyingCorpse 93dc0bdaea934c707c59b2e8c5b973e2ceaa1dc2
2016-08-17 04:08:05 +00:00 by TheFlyingCorpse 4e3da7e3aba59f1ba1aee96c45453141e2e51129
2016-08-17 04:10:41 +00:00 by TheFlyingCorpse 5c0b3c58bd44812b93cec232f35fc06267ca71a8