search

toni-moreno / snmpcollector

A full featured Generic SNMP data collector with Web Administration Interface for InfluxDB
MIT License
290 stars 53 forks source link

Influx database support #1

Closed theo-bot closed 8 years ago

theo-bot commented 8 years ago

Hi

Which influx databases are supported? I currently use 0.13.0. But I don't see any metrics in the database.

Kind regards, Theo

toni-moreno commented 8 years ago

Hi @Mainframe2008 , first I Would tell you this is a very early alfa version of this tool. and there is planned lots of new improvements. But you can testit of course.

I've been working on with influx 0.10 and 0.11, but this tool is made with the last client version and it should be fully compatible with 0.12/0.13 and 1 beta.

Can you put in debug mode and show me the output log? Can you show me your config.toml ?

toni-moreno commented 8 years ago

You can also execute this command. 

./bin/snpmcolector -showconf

It will show you which devices and witch metrics the collector will send to the influxdb backend.

theo-bot commented 8 years ago

[general] logdir = "./log" logLevel = "debug"

[selfmon] enabled = false freq = 30 prefix = "gvm" extratags = [ "instance=snmp1" ]

[metrics] [metrics."ifHCInOctets"] fieldname = "In.bytes" description = "Bytes In - 64-bit Counters" baseOID = ".1.3.6.1.2.1.31.1.1.1.6" DatasrcType = "COUNTER64" getrate = false

[metrics."ifHCOutOctets"] fieldname = "Out.bytes" description = "Bytes In - 64-bit Counters" baseOID = ".1.3.6.1.2.1.31.1.1.1.10" DatasrcType = "COUNTER64" getRate = false

[metrics."ifInOctets"] fieldname = "In.bytes" description = "Bytes In - 32-bit Counters" baseOID = ".1.3.6.1.2.1.2.2.1.10" DatasrcType = "COUNTER32" getRate = false

[metrics."ifOutOctets"] fieldname = "Out.bytes" description = "Bytes Out - 32-bit Counters" baseOID = ".1.3.6.1.2.1.2.2.1.16" DatasrcType = "COUNTER32" getRate = false

[measurements] [measurements."network_32bits"] name = "32bit-ports" fields = [ "ifInOctets", "ifOutOctets" ] getmode = "indexed" indexOID = ".1.3.6.1.2.1.31.1.1.1.1" # ifName => needed to be "STRING" IndexTAG = "portName"

[measurements."network_64bits"] name = "64bit-ports" fields = [ "ifHCInOctets", "ifHCOutOctets" ] getmode = "indexed" indexOID = ".1.3.6.1.2.1.31.1.1.1.1" # ifName => needed to be "STRING" IndexTAG = "portName"

[getgroups]

[getgroups."network32"] measurements = [ "network_32bits" ]

[snmpdevice]

[snmpdevice.centos7-1] host = "127.0.0.1" port = 161 timeout = 20 retries = 5 snmpversion = "2c" community = "public" devicetagname = "server" matricgroups = [ "network32" ]

[influx]

[influx."*"] host = "127.0.0.1" port = 8086 db = "snmp" user = "snmpuser" password = "snmppass"

[http] port = 8090 adminuser = "admin" adminpassword = "access"

INFO[0000] First SNMP connection to host 127.0.0.1 stablished INFO[0000] Got basic system info Linux centos7-1.localdomain 3.10.0-327.18.2.el7.x86_64 #1 SMP Thu May 12 11:03:55 UTC 2016 x86_64 ; UPTIME (5 h30m48s) Root root@localhost (configure /etc/snmp/snmp.local.conf) ; centos7-1.localdomain ; Unknown (edit /etc/snmp/snmpd.conf) ;
INFO[0000] Beginning gather process for device centos7-1 (127.0.0.1) INFO[0000] snmpdevice [centos7-1] snmp poolling took [57ns]
INFO[0000] snmpdevice [centos7-1] influx send took [2.215µs] INFO[0030] snmpdevice [centos7-1] snmp poolling took [406ns]
INFO[0030] snmpdevice [centos7-1] influx send took [3.158µs] INFO[0060] snmpdevice [centos7-1] snmp poolling took [331ns]
INFO[0060] snmpdevice [centos7-1] influx send took [2.105µs] INFO[0090] snmpdevice [centos7-1] snmp poolling took [336ns]
INFO[0090] snmpdevice [centos7-1] influx send took [2.455µs] INFO[0120] snmpdevice [centos7-1] snmp poolling took [341ns]
INFO[0120] snmpdevice [centos7-1] influx send took [2.085µs] INFO[0150] snmpdevice [centos7-1] snmp poolling took [336ns]
INFO[0150] snmpdevice [centos7-1] influx send took [2.22µs]

[influxdb@centos7-1 snmpcollector]$ ./bin/snmpcollector -showconf INFO[0000] set Default directories :

DEBU[0000] --------------------Initializing Config metrics------------------- DEBU[0000] Initializing SNMPMetricconfig...
DEBU[0000] Initializing MEASSUREMENTSconfig...
INFO[0000] processing measurement key: network_32bits
DEBU[0000] %+v&{network_32bits 32bit-ports [ifInOctets ifOutOctets] indexed .1.3.6.1.2.1.31.1.1.1.1 portName []} DEBU[0000] looking for measure 32bit-ports fields: ifInOctets DEBU[0000] Found ok!
DEBU[0000] looking for measure 32bit-ports fields: ifOutOctets DEBU[0000] Found ok!
DEBU[0000] FIELDMETRICS: [0xc82013c070 0xc82013c460]
INFO[0000] processing measurement key: network_64bits
DEBU[0000] %+v&{network_64bits 64bit-ports [ifHCInOctets ifHCOutOctets] indexed .1.3.6.1.2.1.31.1.1.1.1 portName []} DEBU[0000] looking for measure 64bit-ports fields: ifHCInOctets DEBU[0000] Found ok!
DEBU[0000] looking for measure 64bit-ports fields: ifHCOutOctets DEBU[0000] Found ok!
DEBU[0000] FIELDMETRICS: [0xc82013c850 0xc82013bc70]
DEBU[0000] -----------------------END Config metrics---------------------- INFO[0000] Initializing device centos7-1

SNMP host: centos7-1

Host: 127.0.0.1 Port: 161 Version: 2c

toni-moreno commented 8 years ago

I think you have a mistake in your config

matricgroups = [ "network32" ]

should be

metricgroups = [ "network32" ]

can you fix and test now?

theo-bot commented 8 years ago

Toni,

I fixed the metricgroup. If the snmpdevice is set to "1" nothing happens. If I set it to "2c" I get a bunch of errors: panic: interface conversion: interface is uint, not int32

goroutine 1 [running]: main.pduVal2Int64(0xc8201c8980, 0x17, 0x41, 0x8a6f80, 0xc8201c7980, 0x1) /apps/influxsnmp/src/github.com/toni-moreno/snmpcollector/pkg/snmp.go:17 +0xb8 main.(_InfluxMeasurement).SnmpBulkData.func1(0xc8201c8980, 0x17, 0xc820124b41, 0x8a6f80, 0xc8201c7980, 0x0, 0x0) /apps/influxsnmp/src/github.com/toni-moreno/snmpcollector/pkg/influxmeasurement.go:223 +0x63a github.com/soniah/gosnmp.(_GoSNMP).walk(0xc820126300, 0x47d1a5, 0xc820124b00, 0x15, 0xc8201371a0, 0x0, 0x0) /apps/influxsnmpd/src/github.com/soniah/gosnmp/walk.go:69 +0x852 github.com/soniah/gosnmp.(_GoSNMP).BulkWalk(0xc820126300, 0xc820124b00, 0x15, 0xc8201371a0, 0x0, 0x0) /apps/influxsnmpd/src/github.com/soniah/gosnmp/gosnmp.go:330 +0x50 main.(_InfluxMeasurement).SnmpBulkData(0xc8201456c0, 0xc820126300, 0xc8201bf090, 0xc820137380, 0x0, 0x0) /apps/influxsnmp/src/github.com/toni-moreno/snmpcollector/pkg/influxmeasurement.go:230 +0x16a main.(_SnmpDeviceCfg).InitDevSnmpInfo(0xc8200da600) /apps/influxsnmp/src/github.com/toni-moreno/snmpcollector/pkg/snmpdevice.go:263 +0x2d23 main.(_SnmpDeviceCfg).Init(0xc8200da600, 0xc820013dc2, 0x9) /apps/influxsnmp/src/github.com/toni-moreno/snmpcollector/pkg/snmpdevice.go:346 +0xaed main.init.1() /apps/influxsnmp/src/github.com/toni-moreno/snmpcollector/pkg/main.go:148 +0x6e0 main.init() /apps/influxsnmp/src/github.com/toni-moreno/snmpcollector/pkg/webserver.go:199 +0x3a8

goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

But that is a differnt issue i guess.

Kind regards,

Theo

toni-moreno commented 8 years ago

I will review this trace and try to fix in next days.

Thank you to report it

toni-moreno commented 8 years ago

Hi @Mainframe2008 I did a commit yesterday that sould fix your crash. Could you pull new commits / compile and test again?

Thx a lot!

theo-bot commented 8 years ago

Toni

I recompiled and tested again. No problems recompiling. With snmp v2c i get negative values:

select * from "32bit-ports" where portName='eno16777736'
name: 32bit-ports
-----------------
time            In.bytes    Out.bytes   portName    server
1467202813236119607 0       0       eno16777736 centos7-1
1467202843237180660 0       0       eno16777736 centos7-1
1467202873236994915 0       0       eno16777736 centos7-1
1467202903236611998 0       0       eno16777736 centos7-1
1467202933236726803 -342        -342        eno16777736 centos7-1
1467202963236427154 -822        -677        eno16777736 centos7-1
1467202993236575788 -433        -180        eno16777736 centos7-1

And no values using snmp v1.

Kind regards

Theo

toni-moreno commented 8 years ago

If you are getting negative values is because snmp is indeed returning a 64 bit counter. Can you change the DatasrcType to COUNTER64 and test again ?

If not sure about bitsize you can just config as INTEGER and the agent wont compute online the increment (as it does when COUNTERXX is configured) you will instead use derivative function in influxdb to get the real load.

I'm not a SNMP guru so I will investigate about Counter32 OID's later.

About snmpv1 i suggest to configure log=debug an send me the output . Please.

Thank you for you help !!!

toni-moreno commented 8 years ago

Hi @Mainframe2008 , I've finally found the problem on the negative values!! It's a silly error on the code, when cooking data. Operatin order are inverted in the 32 bits couters.

BAD: https://github.com/toni-moreno/snmpcollector/blob/master/pkg/snmpmetric.go#L113

OK https://github.com/toni-moreno/snmpcollector/blob/master/pkg/snmpmetric.go#L140

I will build the fix in a few hours...

Sorry for this stupid mistake

toni-moreno commented 8 years ago

Hi @Mainframe2008 the code is now fixed !!!

theo-bot commented 8 years ago

@toni-moreno

Today I had the chance to test. It looks good indeed.

Thanks

Theo

toni-moreno commented 8 years ago

Thank you very much @theo-bot , I will close this issue.

I wish you can continue testing this tool when finished the second big refactor.

We have plans to:

Let me know what of the current and planned features would you like to test.

theo-bot commented 8 years ago

@toni-moreno

Nice developments and I will follow your project for sure. I work for an enterprise level organization and we are looking for tools which can be part of a process chain. That means it should be able to be provisioned by other tools. So it has to have something like a rest interface for example.

Let me know if you have anything to test.

Regards

Theo