Snmp Trap support in snmp export

[cleonte]

Hi Team,

any plan to implement snmp traps in prometheus?

Cheers

[brian-brazil]

I think this would end up as a separate exporter.

In general SNMP traps are quite difficult to work with, and would strongly recommend using data from the snmp_exporter to alert on instead.

[cleonte]

Agree,

snmp traps kinda hard to be handled... almost all open-source monitoring are ignoring it or doing some hacks around it...(icinga2/nagios)/ bosun...

[RichiH]

Random data point: Running large networks for about a decade now, I have never needed traps. They might be a tad quicker, but usually don't tell you anything you can't query yourself. The cost of special casing for a completely distinct path of information is higher than the benefit of events which can easily be lost during network events, imo.

[brian-brazil]

There's no plans to support this.

[Anand-Chandan]

I can see that there was no plan to support snmp traps in snmp exporter as per last update. Is it still same or does it support now or is there any future plan?

[xkilian]

Snmp traps should still be handled by snmptrapd + Snmptt + Rsyslogd/Syslogng + promtail + Loki (or splunk).

It still handles and formats them correctly, and the Snmptt files can be managed for severity.. I tried Nxlog which in a single pass extracts things in rfc5424. Super nice. but... then it bogs down in Regex hell for the extra values... Not as flexible and mature as the Snmptt conf files. Snmp exporter should Never ever have traps in scope.

The big win would be to have Snmptt improved to send data in RFC5424 directly to rsyslog with some type of wal. Now THAT would be awesome.

[xkilian]

And more improvements on syslog fields when sending the log. Host,app, time stamp, etc. Sadly Snmptt is no longer actively developed. Nice summer project for an intern though..

[RichiH]

For reference, this the current thinking, copied from https://docs.google.com/document/d/1McJJIiJfHgoecVrVNXx4ABJmI5M21e-6O9IgMNbVnvw/edit#heading=h.90jmffx73fn7

Outside of dying gasps, SNMP traps tend to be a worse engineering trade-off than SNMP walk/bulk get/get. Yet, they are part of how many organizations work. As traps share many characteristics of events, sending them to a system like Grafana Loki and extracting metrics from there seems to be the best design overall. This could happen through two ways

• SNMP trap -> syslog -> promtail -> Loki
• A native SNMP trap -> Loki “exporter”

[xkilian]

In the second case, you are essentially rewriting Snmptt and potentially snmptrapd and still expose the output in syslogish/rfc5424. A big endeavor for little new value.

I stand by my analysis that the most efficient and best approach is minor improvements to Snmptt to turn it into a Loki/Promtail data source. Labels, rfc5424, flexibility in the base fields. Alternate simpler output for unknown traps.

I think both proposals you raised and I highlighted are one and the same. ;-) (minus the Go maintainability, but much less engineering time required.)

Cheers

[RichiH]

Agreed it's all roughly the same; I was just trying to document. My preference is the first one, as it's less work, and taking in data from domain-specific tooling means someone else found all the corner cases for you already.

If anyone wants to write a short howto, I would be more than happy to make it part of the official snmp_exporter documentation and close the topic in the linked doc.

[xkilian]

I will write something up, I have just gone through the process recently for a new environment so it is fresh in my memory. I even have a nice draw.io diagram that I can adapt.

[RichiH]

Thanks!

[xkilian]

I also left a message on the snmptt dev mailing list with a few improvement suggestions. As he is currently working on 1.5.