search

prometheus / snmp_exporter

SNMP Exporter for Prometheus
Apache License 2.0
1.72k stars 631 forks source link

multiline string in SNMPv2-MIB:sysDescr produces garbled output #440

Closed localpref closed 5 years ago

localpref commented 5 years ago

A multiline sysDescr string on an ancient Cisco 2800 produces garbled output and no valid key{tags} value line, breaking the Prometheus HTTP Metrics output format.

Host operating system: output of uname -a

Linux (hostname) 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3.1 (2019-02-19) x86_64 GNU/Linux (added info: this is running within an LXC container)

snmp_exporter version: output of snmp_exporter -version

snmp_exporter, version 0.15.0 (branch: HEAD, revision: 92a3da4467f8bc6759cf197e7442b8c43e890b13) build user: root@8c3a7c03d455 build date: 20190212-13:40:02 go version: go1.11.5 (same error also with go 1.2.6)

What device/snmpwalk OID are you using?

1.3.6.1.2.1.1.1 (sysDescr)

relevant excerpt from snmp.yml:

sysdescr_test:
  get:
  - 1.3.6.1.2.1.1.1.0
  metrics:
  - name: sysDescr
    oid: 1.3.6.1.2.1.1.1
    type: DisplayString
    help: A textual description of the entity - 1.3.6.1.2.1.1.1
  version: 2
  max_repetitions: 15
  timeout: 10s
  auth:
    community: public

If this is a new device, please link to the MIB(s).

It's an ancient Cisco2800, but the query goes to SNMPv2-MIB::sysDescr

What did you do that produced an error?

Query a sysDescr that is multiline

What did you expect to see?

[...]
# HELP sysDescr A textual description of the entity - 1.3.6.1.2.1.1.1
# TYPE sysDescr gauge
sysDescr{sysDescr="Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M3, RELEASE SOFTWARE (fc1)\nTechnical Support: http://www.cisco.com/techsupport\nCopyright (c) 1986-2011 by Cisco Systems, Inc.\nCompiled Tue 06-Dec-11 16:21 by prod_rel_team\n"} 1
# snmpbulkwalk -On -v2c -c public (ip) 1.3.6.1.2.1.1.1
.1.3.6.1.2.1.1.1.0 = STRING: Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Tue 06-Dec-11 16:21 by prod_rel_team

What did you see instead?

[...]
# HELP sysDescr A textual description of the entity - 1.3.6.1.2.1.1.1
# TYPE sysDescr gauge
\nCompiled Tue 06-Dec-11 16:21 by prod_rel_team"} 1rt(C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M3, RELEASE SOFTWARE (fc1)
localpref commented 5 years ago

I see the same behaviour with a manually built snmp_exporter from a few days ago( HEAD/40d8840915d961a8e3d3ae88a65f5d35072a9d8b), built with go 1.2.6

brian-brazil commented 5 years ago

Something very odd is going on here, invalid output like this should be impossible. Can you get the hex version of both the sysDescr and what's on the /metrics?

localpref commented 5 years ago

(small omissions of internal data, as bulkwalk returned more objects than necessary)

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:12:04.437905 IP mgr > device:  GetBulk(27)  N=0 M=10 .1.3.6.1.2.1.1.1
    0x0000:  4500 0046 f26a 4000 4011 d386 a9fe fd0d  E..F.j@.@.......
    0x0010:  c0a8 0d01 85c4 00a1 0032 74f9 3028 0201  .........2t.0(..
    0x0020:  0104 0670 7562 6c69 63a5 1b02 046e 161c  ...public....n..
    0x0030:  8602 0100 0201 0a30 0d30 0b06 072b 0601  .......0.0...+..
    0x0040:  0201 0101 0500                           ......
15:12:04.442150 IP device > mgr:  GetResponse(478)  .1.3.6.1.2.1.1.1.0=43_69_73_63_6f_20_49_4f_53_20_53_6f_66_74_77_61_72_65_2c_20_32_38_30_30_20_53_6f_66_74_77_61_72_65_20_28_43_32_38_30_30_4e_4d_2d_41_44_56_45_4e_54_45_52_50_52_49_53_45_4b_39_2d_4d_29_2c_20_56_65_72_73_69_6f_6e_20_31_35_2e_31_28_34_29_4d_33_2c_20_52_45_4c_45_41_53_45_20_53_4f_46_54_57_41_52_45_20_28_66_63_31_29_0d_0a_54_65_63_68_6e_69_63_61_6c_20_53_75_70_70_6f_72_74_3a_20_68_74_74_70_3a_2f_2f_77_77_77_2e_63_69_73_63_6f_2e_63_6f_6d_2f_74_65_63_68_73_75_70_70_6f_72_74_0d_0a_43_6f_70_79_72_69_67_68_74_20_28_63_29_20_31_39_38_36_2d_32_30_31_31_20_62_79_20_43_69_73_63_6f_20_53_79_73_74_65_6d_73_2c_20_49_6e_63_2e_0d_0a_43_6f_6d_70_69_6c_65_64_20_54_75_65_20_30_36_2d_44_65_63_2d_31_31_20_31_36_3a_32_31_20_62_79_20_70_72_6f_64_5f_72_65_6c_5f_74_65_61_6d 
    0x0000:  4500 020d 4349 0000 fd11 03e1 c0a8 0d01  E...CI..........
    0x0010:  a9fe fd0d 00a1 85c4 01f9 37f1 3082 01ed  ..........7.0...
    0x0020:  0201 0104 0670 7562 6c69 63a2 8201 de02  .....public.....
    0x0030:  046e 161c 8602 0100 0201 0030 8201 ce30  .n.........0...0
    0x0040:  8201 0906 082b 0601 0201 0101 0004 81fc  .....+..........
    0x0050:  4369 7363 6f20 494f 5320 536f 6674 7761  Cisco.IOS.Softwa
    0x0060:  7265 2c20 3238 3030 2053 6f66 7477 6172  re,.2800.Softwar
    0x0070:  6520 2843 3238 3030 4e4d 2d41 4456 454e  e.(C2800NM-ADVEN
    0x0080:  5445 5250 5249 5345 4b39 2d4d 292c 2056  TERPRISEK9-M),.V
    0x0090:  6572 7369 6f6e 2031 352e 3128 3429 4d33  ersion.15.1(4)M3
    0x00a0:  2c20 5245 4c45 4153 4520 534f 4654 5741  ,.RELEASE.SOFTWA
    0x00b0:  5245 2028 6663 3129 0d0a 5465 6368 6e69  RE.(fc1)..Techni
    0x00c0:  6361 6c20 5375 7070 6f72 743a 2068 7474  cal.Support:.htt
    0x00d0:  703a 2f2f 7777 772e 6369 7363 6f2e 636f  p://www.cisco.co
    0x00e0:  6d2f 7465 6368 7375 7070 6f72 740d 0a43  m/techsupport..C
    0x00f0:  6f70 7972 6967 6874 2028 6329 2031 3938  opyright.(c).198
    0x0100:  362d 3230 3131 2062 7920 4369 7363 6f20  6-2011.by.Cisco.
    0x0110:  5379 7374 656d 732c 2049 6e63 2e0d 0a43  Systems,.Inc...C
    0x0120:  6f6d 7069 6c65 6420 5475 6520 3036 2d44  ompiled.Tue.06-D
    0x0130:  6563 2d31 3120 3136 3a32 3120 6279 2070  ec-11.16:21.by.p
    0x0140:  726f 645f 7265 6c5f 7465 616d 3015 0608  rod_rel_team0...
    0x0150:  2b06 0102 0101 0200 0609 2b06 0104 0109  +.........+.....
    0x0160:  0184 4130 1106 082b 0601 0201 0103 0043  ..A0...+.......C
    0x0170:  0500 91c0 4b51 3013 0608 2b06 0102 0101  ....KQ0...+.....

curl | hexdump -C

00000180  2d 2d 3a 2d 2d 20 32 31  32 38 32 0a 23 20 48 45  |--:-- 21282.# HE|
00000190  4c 50 20 73 6e 6d 70 5f  73 63 72 61 70 65 5f 64  |LP snmp_scrape_d|
000001a0  75 72 61 74 69 6f 6e 5f  73 65 63 6f 6e 64 73 20  |uration_seconds |
000001b0  54 6f 74 61 6c 20 53 4e  4d 50 20 74 69 6d 65 20  |Total SNMP time |
000001c0  73 63 72 61 70 65 20 74  6f 6f 6b 20 28 77 61 6c  |scrape took (wal|
000001d0  6b 20 61 6e 64 20 70 72  6f 63 65 73 73 69 6e 67  |k and processing|
000001e0  29 2e 0a 23 20 54 59 50  45 20 73 6e 6d 70 5f 73  |)..# TYPE snmp_s|
000001f0  63 72 61 70 65 5f 64 75  72 61 74 69 6f 6e 5f 73  |crape_duration_s|
00000200  65 63 6f 6e 64 73 20 67  61 75 67 65 0a 73 6e 6d  |econds gauge.snm|
00000210  70 5f 73 63 72 61 70 65  5f 64 75 72 61 74 69 6f  |p_scrape_duratio|
00000220  6e 5f 73 65 63 6f 6e 64  73 20 30 2e 30 30 34 34  |n_seconds 0.0044|
00000230  33 37 39 30 39 0a 23 20  48 45 4c 50 20 73 6e 6d  |37909.# HELP snm|
00000240  70 5f 73 63 72 61 70 65  5f 70 64 75 73 5f 72 65  |p_scrape_pdus_re|
00000250  74 75 72 6e 65 64 20 50  44 55 73 20 72 65 74 75  |turned PDUs retu|
00000260  72 6e 65 64 20 66 72 6f  6d 20 77 61 6c 6b 2e 0a  |rned from walk..|
00000270  23 20 54 59 50 45 20 73  6e 6d 70 5f 73 63 72 61  |# TYPE snmp_scra|
00000280  70 65 5f 70 64 75 73 5f  72 65 74 75 72 6e 65 64  |pe_pdus_returned|
00000290  20 67 61 75 67 65 0a 73  6e 6d 70 5f 73 63 72 61  | gauge.snmp_scra|
000002a0  70 65 5f 70 64 75 73 5f  72 65 74 75 72 6e 65 64  |pe_pdus_returned|
000002b0  20 31 0a 23 20 48 45 4c  50 20 73 6e 6d 70 5f 73  | 1.# HELP snmp_s|
000002c0  63 72 61 70 65 5f 77 61  6c 6b 5f 64 75 72 61 74  |crape_walk_durat|
000002d0  69 6f 6e 5f 73 65 63 6f  6e 64 73 20 54 69 6d 65  |ion_seconds Time|
000002e0  20 53 4e 4d 50 20 77 61  6c 6b 2f 62 75 6c 6b 77  | SNMP walk/bulkw|
000002f0  61 6c 6b 20 74 6f 6f 6b  2e 0a 23 20 54 59 50 45  |alk took..# TYPE|
00000300  20 73 6e 6d 70 5f 73 63  72 61 70 65 5f 77 61 6c  | snmp_scrape_wal|
00000310  6b 5f 64 75 72 61 74 69  6f 6e 5f 73 65 63 6f 6e  |k_duration_secon|
00000320  64 73 20 67 61 75 67 65  0a 73 6e 6d 70 5f 73 63  |ds gauge.snmp_sc|
00000330  72 61 70 65 5f 77 61 6c  6b 5f 64 75 72 61 74 69  |rape_walk_durati|
00000340  6f 6e 5f 73 65 63 6f 6e  64 73 20 30 2e 30 30 34  |on_seconds 0.004|
00000350  33 34 37 30 33 37 0a 23  20 48 45 4c 50 20 73 79  |347037.# HELP sy|
00000360  73 44 65 73 63 72 20 41  20 74 65 78 74 75 61 6c  |sDescr A textual|
00000370  20 64 65 73 63 72 69 70  74 69 6f 6e 20 6f 66 20  | description of |
00000380  74 68 65 20 65 6e 74 69  74 79 20 2d 20 31 2e 33  |the entity - 1.3|
00000390  2e 36 2e 31 2e 32 2e 31  2e 31 2e 31 0a 23 20 54  |.6.1.2.1.1.1.# T|
000003a0  59 50 45 20 73 79 73 44  65 73 63 72 20 67 61 75  |YPE sysDescr gau|
000003b0  67 65 0a 73 79 73 44 65  73 63 72 7b 73 79 73 44  |ge.sysDescr{sysD|
000003c0  65 73 63 72 3d 22 43 69  73 63 6f 20 49 4f 53 20  |escr="Cisco IOS |
000003d0  53 6f 66 74 77 61 72 65  2c 20 32 38 30 30 20 53  |Software, 2800 S|
000003e0  6f 66 74 77 61 72 65 20  28 43 32 38 30 30 4e 4d  |oftware (C2800NM|
000003f0  2d 41 44 56 45 4e 54 45  52 50 52 49 53 45 4b 39  |-ADVENTERPRISEK9|
00000400  2d 4d 29 2c 20 56 65 72  73 69 6f 6e 20 31 35 2e  |-M), Version 15.|
00000410  31 28 34 29 4d 33 2c 20  52 45 4c 45 41 53 45 20  |1(4)M3, RELEASE |
00000420  53 4f 46 54 57 41 52 45  20 28 66 63 31 29 0d 5c  |SOFTWARE (fc1).\|
00000430  6e 54 65 63 68 6e 69 63  61 6c 20 53 75 70 70 6f  |nTechnical Suppo|
00000440  72 74 3a 20 68 74 74 70  3a 2f 2f 77 77 77 2e 63  |rt: http://www.c|
00000450  69 73 63 6f 2e 63 6f 6d  2f 74 65 63 68 73 75 70  |isco.com/techsup|
00000460  70 6f 72 74 0d 5c 6e 43  6f 70 79 72 69 67 68 74  |port.\nCopyright|
00000470  20 28 63 29 20 31 39 38  36 2d 32 30 31 31 20 62  | (c) 1986-2011 b|
00000480  79 20 43 69 73 63 6f 20  53 79 73 74 65 6d 73 2c  |y Cisco Systems,|
00000490  20 49 6e 63 2e 0d 5c 6e  43 6f 6d 70 69 6c 65 64  | Inc..\nCompiled|
000004a0  20 54 75 65 20 30 36 2d  44 65 63 2d 31 31 20 31  | Tue 06-Dec-11 1|
000004b0  36 3a 32 31 20 62 79 20  70 72 6f 64 5f 72 65 6c  |6:21 by prod_rel|
000004c0  5f 74 65 61 6d 22 7d 20  31 0a                    |_team"} 1.|
localpref commented 5 years ago

hmm, is that some weird \LF vs \CRLF and the shell etc overwriting itself is going on?

localpref commented 5 years ago

the device sends the 0x0d character before every newline, which is "\CR" or "Carriage Return" but no linefeed...

localpref commented 5 years ago 
curl "http://localhost:9117/snmp?module=sysdescr_public&target=192.168.13.1" | tr -d '\r'
[...]
# HELP sysDescr A textual description of the entity - 1.3.6.1.2.1.1.1
# TYPE sysDescr gauge
sysDescr{sysDescr="Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M3, RELEASE SOFTWARE (fc1)\nTechnical Support: http://www.cisco.com/techsupport\nCopyright (c) 1986-2011 by Cisco Systems, Inc.\nCompiled Tue 06-Dec-11 16:21 by prod_rel_team"} 1
localpref commented 5 years ago

I think you can close this, this smells of "user too stupid" error. I don't think theres anything wrong with the exporter.

localpref commented 5 years ago

hmm, as you do escape \n, do you also need to escape \r in the exporter?

localpref commented 5 years ago

TLDR: Old Cisco device sends DOS-Style newlines with \r\n. snmp_exporter escapes the \n newline, but does not escape the \r carriage return, causing the output to overwrite itself.

brian-brazil commented 5 years ago

False alarm then.

localpref commented 5 years ago

Hmmm. The device sends 0d 0a (\r \n) (see tcpdump), but the /metrics output gives 0d 5c 6e (\r \\n), i.e. an escaped newline. I think that is actually a problem - the exporter should escape both control characters or none.

tcpdump / on the wire
0x00d0:  703a 2f2f 7777 772e 6369 7363 6f2e 636f  p://www.cisco.co
0x00e0:  6d2f 7465 6368 7375 7070 6f72 740d 0a43  m/techsupport..C
                                         _____
output of exporter http /snmp
00000450  69 73 63 6f 2e 63 6f 6d  2f 74 65 63 68 73 75 70  |isco.com/techsup|
00000460  70 6f 72 74 0d 5c 6e 43  6f 70 79 72 69 67 68 74  |port.\nCopyright|
                      ________
brian-brazil commented 5 years ago

That's the correct encoding for a /metrics.