search

prometheus / snmp_exporter

SNMP Exporter for Prometheus
Apache License 2.0
1.66k stars 615 forks source link

Generator detects debug messages from NetSNMP as parsing errors #508

Open pleonex opened 4 years ago

pleonex commented 4 years ago

Host operating system

Statically compiled from Docker image golang:1.14.2-alpine3.11 and run in Docker image prom/snmp-exporter:v0.17.0. Compiled as follow:

FROM golang:1.14.2-alpine3.11 as stage

RUN apk add musl-dev gcc git net-snmp-dev curl openssl-libs-static && \
    go get -d -v github.com/prometheus/snmp_exporter/generator && \
    cd /go/src/github.com/prometheus/snmp_exporter/generator && \
    git checkout v0.17.0 && \
    go build -a --ldflags '-extldflags "-lcrypto -static"' -o snmp_generator *.go && \
    cp snmp_generator /go/bin/

snmp_exporter version

v0.17.0

What device/snmpwalk OID are you using?

N/A

If this is a new device, please link to the MIB(s).

N/A

What did you do that produced an error?

Run the generator as follow:

/bin/snmp_generator --fail-on-parse-errors --log.level=debug generate

What did you expect to see?

No error parsing error message, the operation should have succeed.

What did you see instead?

The process returned with error code 1 and showing this warning:

level=warn ts=2020-04-24T22:18:34.543Z caller=main.go:120 msg="NetSNMP reported parse error(s)" errors=4

After adding in main.go at line 120 the following line: fmt.Println(parseOutput) I saw the reason of the failure:

level=warn ts=2020-04-24T22:18:34.543Z caller=main.go:120 msg="NetSNMP reported parse error(s)" errors=4
Created directory: /var/lib
Created directory: /var/lib/net-snmp
Created directory: /var/lib/net-snmp/mib_indexes
MIB search path: /etc/snmp_exporter/mibs/

NetSNMP is printing these four debug messages and the generator thinks they are parsing errors since it just counts the number of output lines from NetSNMP.

I also suggest that the generator prints always as debug level of verbosity the output of NetSNMP so it's easier to debug without code changes.

brian-brazil commented 4 years ago

Why is netsnmp printing those? I've never seen them when I'm running it.

pleonex commented 4 years ago

I was reading a bit the code of NET-SNMP. It seems that the warning about Created directory: only shows if those directories don't exist. So just by creating them in my Dockerfile it stopped showing.

Regarding the MIB search path warning, it shows when there are MIB modules not found (snmplib/parse.c line 778). However, I am missing the actual warning about which module cannot be found (so I can add it). It seems that NET-SNMP only shows these not found MIB modules warnings if you enable a setting from a configuration file. However I haven't been able to enable that. I tried creating a file in /usr/share/snmp/snmp.conf with this content:

dodebugging  1
showmiberrors  yes
mibwarninglevel  2

I created the file using the command snmpconf. Maybe something needs to be done at the level of the code to read the configuration file / enable that warning level?

brian-brazil commented 4 years ago

I don't know offhand, however I'd expect this to Just Work on the part of net-snmp as we're not doing anything unusual.

pleonex commented 4 years ago

I think I found the issue. NET-SNMP reads the configuration files via the function read_configs(). This function is called when initializing the library with void init_snmp(const char *type) (ref), where type is an identifier of the application that uses the library. However, the snmp exporter generator doesn't initialize the library in this way, instead it calls: netsnmp_init_mib() (ref).

It seems the recommended way to initialize the application is via init_snmp as it's shown in their tutorial here. init_snmp calls internally also netsmp_init_mib().

Then, NET-SNMP reads configuration files from several places but one of them is via reading the environment variable SNMPCONFPATH that should point to a path (or list of paths) with the configuration files.

I tried and now I get all the verbosity output I was expecting with my configuration file. If you agree with that change, I can make a PR.

My warnings

``` snmp-exporter_1 | Importing enterprises from replacement module SNMPv2-SMI instead of RFC1155-SMI (/etc/snmp_exporter/mibs//powernet432.mib) snmp-exporter_1 | Loading replacement module SNMPv2-SMI for RFC1213-MIB (/etc/snmp_exporter/mibs//powernet432.mib) snmp-exporter_1 | Did not find 'DisplayString' in module #-1 (/etc/snmp_exporter/mibs//powernet432.mib) snmp-exporter_1 | MIB search path: /etc/snmp_exporter/mibs/ snmp-exporter_1 | Cannot find module (IANAifType-MIB): At line 13 in /etc/snmp_exporter/mibs//IF-MIB.my snmp-exporter_1 | Did not find 'IANAifType' in module #-1 (/etc/snmp_exporter/mibs//IF-MIB.my) snmp-exporter_1 | Cannot find module (IANAifType-MIB): At line 39 in /etc/snmp_exporter/mibs//ISDN-MIB.my snmp-exporter_1 | Importing transmission from replacement module SNMPv2-SMI instead of RFC1213-MIB (/etc/snmp_exporter/mibs//ISDN-MIB.my) snmp-exporter_1 | Did not find 'IANAifType' in module #-1 (/etc/snmp_exporter/mibs//ISDN-MIB.my) snmp-exporter_1 | Cannot find module (SNMP-FRAMEWORK-MIB): At line 26 in /etc/snmp_exporter/mibs//ENTITY-MIB.my snmp-exporter_1 | Did not find 'SnmpAdminString' in module #-1 (/etc/snmp_exporter/mibs//ENTITY-MIB.my) snmp-exporter_1 | Cannot find module (SNMP-FRAMEWORK-MIB): At line 14 in /etc/snmp_exporter/mibs//ENTITY-SENSOR-MIB.my snmp-exporter_1 | Did not find 'SnmpAdminString' in module #-1 (/etc/snmp_exporter/mibs//ENTITY-SENSOR-MIB.my) snmp-exporter_1 | Cannot find module (RMON-MIB): At line 48 in /etc/snmp_exporter/mibs//CISCO-VTP-MIB.my snmp-exporter_1 | Cannot find module (SNMP-FRAMEWORK-MIB): At line 50 in /etc/snmp_exporter/mibs//CISCO-VTP-MIB.my snmp-exporter_1 | Did not find 'SnmpAdminString' in module #-1 (/etc/snmp_exporter/mibs//CISCO-VTP-MIB.my) ```

brian-brazil commented 4 years ago

That seems reasonable, how would we override things though to ensure that only parse errors show up?

pleonex commented 4 years ago

Apart from the log message saying that it will create a directory, the rest of the messages are valid warnings or errors. I enabled the logging and tried with the MIB files downloaded via make mibs and there are several warnings already reported about those MIBs:

level=warn ts=2020-06-11T13:46:37.008Z caller=main.go:120 msg="NetSNMP reported parse error(s)" errors=16
MODULE-IDENTITY MACRO (lines 55..79 parsed and ignored).
OBJECT-IDENTITY MACRO (lines 81..103 parsed and ignored).
OBJECT-TYPE MACRO (lines 212..298 parsed and ignored).
NOTIFICATION-TYPE MACRO (lines 302..334 parsed and ignored).
TEXTUAL-CONVENTION MACRO (lines 8..48 parsed and ignored).
Importing transmission from replacement module SNMPv2-SMI instead of RFC1213-MIB (mibs/ISDN-MIB)
Importing enterprises from replacement module SNMPv2-SMI instead of RFC1155-SMI (mibs/apc-powernet-mib)
Loading replacement module SNMPv2-SMI for RFC1213-MIB (mibs/apc-powernet-mib)
Importing enterprises from replacement module SNMPv2-SMI instead of RFC1155-SMI (mibs/PICO-SMI-MIB.txt)
Warning: picoMobileMIBNotificationPrefix.0 is both picoMobileMIBNotificationPrefix# and picoMobileMIBNotifications (mibs/PICO-SMI-MIB.txt)
Warning: picoPostMIBNotificationPrefix.0 is both picoPostMIBNotificationPrefix# and picoPostMIBNotifications (mibs/PICO-SMI-MIB.txt)
Warning: picoIsdnMIBNotificationPrefix.0 is both picoIsdnMIBNotificationPrefix# and picoIsdnMIBNotifications (mibs/PICO-SMI-MIB.txt)
Warning: picoNetmonMIBNotificationPrefix.0 is both picoNetmonMIBNotificationPrefix# and picoNetmonMIBNotifications (mibs/PICO-SMI-MIB.txt)
Importing enterprises from replacement module SNMPv2-SMI instead of RFC1155-SMI (mibs/PICO-IPSEC-FLOW-MONITOR-MIB.txt)
Warning: pipSecMIBNotificationPrefix.0 is both pipSecMIBNotificationPrefix# and pipSecMIBNotifications (mibs/PICO-IPSEC-FLOW-MONITOR-MIB.txt)
Importing enterprises from replacement module SNMPv2-SMI instead of RFC1155-SMI (mibs/PICO-SMI-ID-MIB.txt)

To fix the warnings related to RFC1155-SMI and RFC1213 it's just to download the MIB defined in the RFC (unfortunately there are no official download as the official thing is the spec itself with the MIB definition). Not sure about how to fix the warnings related to pico. To fix the first warnings about the macro it would be necessary to find the MIB redefining these macros that already defined in the MIB compilers and remove them.

I think those warnings need to be fixed before enabling this logging.

brian-brazil commented 4 years ago

Generally warnings can be fixed by having exactly the right set of MIBs. Are any of those useful in terms of pointing out problems, or are they noise?

pleonex commented 4 years ago

In this case, looks more than warnings that are nice to fix but not mandatory. But with the set of MIB our software uses, these new warnings help me to realize we were missing MIBs like IANAifType that was causing some metrics to just not appear without any other warning.

brian-brazil commented 4 years ago

If we're to enable a higher level of warnings, then we should first ensure that the example config produces no such warnings.

llamafilm commented 8 months ago

How did you manage to enable this extra verbosity in the generator? I always run snmp-generator in Docker so I tried adding the SNMPCONFPATH variable and the file as shown in this thread but it didn't work. My command is like this:

docker run --rm -it --name snmp-generator \
    -v "${PWD}:/opt/" \
    -v "${PWD}/../mibs:/mibs:ro" \
    -e MIBDIRS=/mibs \
    -e SNMPCONFPATH=/opt/snmp.conf \
    prom/snmp-generator:main generate

I'm trying to diagnose the cause of this error:

caller=main.go:124 level=warn msg="NetSNMP reported parse error(s)" errors=1