GeoIP - Erratic processing log data - NoClassDefFoundError com.fasterxml.jackson.databind.node

[avongluck-r1soft]

Erratic parsing of log data. Huge number of errors seen in the graylog server log file. (looks like almost one error per inbound log event)

Service was running for around a month before this problem started.

Current Behavior

2016-12-27T16:07:07.980-06:00 WARN  [ProcessBuffer] Unable to process event MessageEvent{raw=null, message=source: OMIT | message: 2016/12/27 16:06:17.220247 route.go:268: OMIT:46210 - [][][] - 200 - 1.401199ms - OMIT | go_script: route.go | level: 6 | gl2_remote_ip: OMIT | http_response_time: 1.401199 | gl2_remote_port: 53947 | src_ipv4: OMIT | uuid_generic: OMIT | gl2_source_input: 57d7243c0ed1c946f829fd18 | gl2_source_node: bca2dfe8-77c8-4d01-8495-OMIT | _id: cedc19b1-cc80-11e6-99e0-OMIT | facility: syslogd | timestamp: 2016-12-27T16:06:17.220-06:00 | http_response_code: 200 }, messages=null}, sequence 1764298396
java.lang.NoClassDefFoundError: Could not initialize class com.fasterxml.jackson.databind.node.BinaryNode
    at com.maxmind.db.Decoder.decodeByType(Decoder.java:166) ~[?:?]
    at com.maxmind.db.Decoder.decode(Decoder.java:147) ~[?:?]
    at com.maxmind.db.Decoder.decode(Decoder.java:87) ~[?:?]
    at com.maxmind.db.Reader.resolveDataPointer(Reader.java:252) ~[?:?]
    at com.maxmind.db.Reader.get(Reader.java:150) ~[?:?]
    at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:155) ~[?:?]
    at com.maxmind.geoip2.DatabaseReader.city(DatabaseReader.java:206) ~[?:?]
    at org.graylog.plugins.map.geoip.GeoIpResolverEngine.extractGeoLocationInformation(GeoIpResolverEngine.java:99) ~[?:?]
    at org.graylog.plugins.map.geoip.GeoIpResolverEngine.filter(GeoIpResolverEngine.java:77) ~[?:?]
    at org.graylog.plugins.map.geoip.processor.GeoIpProcessor.process(GeoIpProcessor.java:79) ~[?:?]
    at org.graylog2.buffers.processors.ServerProcessBufferProcessor.handleMessage(ServerProcessBufferProcessor.java:56) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:82) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:61) ~[graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:35) ~[graylog.jar:?]
    at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
    at java.lang.Thread.run(Unknown Source) [?:1.8.0_101]

Possible Solution

Restart graylog-server process.

Steps to Reproduce (for bugs)

Graylog-server running for ~2months

Your Environment

java version "1.8.0_101" Java(TM) SE Runtime Environment (build 1.8.0_101-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.101-b13, mixed mode)

• Graylog Version: 2.1.2
• Operating System: CentOS 7

[avongluck-r1soft]

I just noticed the GeoIP in the trace file. This is the second time we have seen odd processing from GeoIP. I've disabled it in our infrastructure for now. Sounds like a bug? We do around 1,000 eps

[bernd]

Which version of the GeoIP plugin do you use?

[avongluck-r1soft]

I thought GeoIP was built-in?

Here are our plugins...

-rw-r--r--  1 root root   26322 Nov  4 11:09 graylog-plugin-beats-1.1.3.jar
-rw-r--r--  1 root root 2801943 Nov  4 11:09 graylog-plugin-collector-1.1.2.jar
-rw-r--r--  1 root root 2383687 Nov  4 11:09 graylog-plugin-enterprise-integration-1.1.2.jar
-rw-r--r--  1 root root 5753431 Nov  4 11:09 graylog-plugin-map-widget-1.1.2.jar
-rw-r--r--  1 root root 5104528 Nov  4 11:09 graylog-plugin-pipeline-processor-1.1.2.jar
-rw-r--r--  1 root root   23922 Feb 22  2016 graylog-plugin-slack-2.1.0.jar
-rw-r--r--  1 root root 2326122 Oct 21 09:31 opsgenie-graylog-alarmcallback-1.1.0.jar
-rw-r--r--  1 root root  479801 Nov  4 11:09 usage-statistics-2.1.2.jar

[jalogisch]

@avongluck-r1soft Did you notice any other errors in your Graylog server log files?

It would be nice if you could try to run Graylog without the OpsGenie plugin, as this is written for Graylog 1.x and might not be compatible with Graylog 2.x.

If the error is still present, please add a comment to this issue.

[avongluck-r1soft]

We haven't had any additional crashes after disabling the GeoIP plugin. We really can't disable OpsGenie since it is a core part of our event notification pipeline.

[joschi]

@avongluck-r1soft You should still be aware that the OpsGenie plugin on the Graylog Marketplace has been written for Graylog 1.x and could stop working with any update of Graylog.

I'm closing this issue since we were unable to reproduce the error.