GeoIP message processor doesn't reload database files

[bernd]

Expected Behavior

The GeoIP message processor (GeoIpResolverEngine class) should reload the database files when they change.

Current Behavior

The GeoIpResolverEngine loads the database files only once at startup and doesn't re-open the database handles when a database file gets updated.

Possible Solution

Introduce a reload mechanism similar to how MaxMindDataAdapter#refresh works.

The code should also handle in-place database file updates correctly and gracefully handle any errors. (see Context below)

Steps to Reproduce (for bugs)

1. Configure the "Geo-Location Processor" on System/Configurations with a database file and enable it
2. Check that messages with IP address fields have GeoIP fields
3. Copy a new GeoIP version of the database file to the existing location
4. Observe the exception (see Context below)

Context

@drewmiranda-gl noticed the missing reload behavior while updating GeoIP databases. The code raised the following exception when a database file got replaced with the cp command. We should handle such cases and reload the database handle.

2022-07-18T14:19:58.734-05:00 WARN  [MaxMindIpResolver] Error getting ASN for IP Address '/<redacted>'. The MaxMind DB file's data section contains bad data: pointer larger than the database.
com.maxmind.db.InvalidDatabaseException: The MaxMind DB file's data section contains bad data: pointer larger than the database.
    at com.maxmind.db.Decoder.decode(Decoder.java:81) ~[graylog.jar:?]
    at com.maxmind.db.Decoder$1.load(Decoder.java:75) ~[graylog.jar:?]
    at com.maxmind.db.NoCache.get(NoCache.java:19) ~[graylog.jar:?]
    at com.maxmind.db.Decoder.decode(Decoder.java:111) ~[graylog.jar:?]
    at com.maxmind.db.Decoder.decode(Decoder.java:87) ~[graylog.jar:?]
    at com.maxmind.db.Reader.resolveDataPointer(Reader.java:252) ~[graylog.jar:?]
    at com.maxmind.db.Reader.get(Reader.java:150) ~[graylog.jar:?]
    at com.maxmind.geoip2.DatabaseReader.get(DatabaseReader.java:182) ~[graylog.jar:?]
    at com.maxmind.geoip2.DatabaseReader.asn(DatabaseReader.java:261) ~[graylog.jar:?]
    at org.graylog.plugins.map.geoip.MaxMindIpAsnResolver.doGetGeoIpData(MaxMindIpAsnResolver.java:48) [graylog.jar:?]
    at org.graylog.plugins.map.geoip.GeoIpResolver.getGeoIpData(GeoIpResolver.java:64) [graylog.jar:?]
    at org.graylog.plugins.map.geoip.GeoIpResolverEngine.addGeoIpDataIfPresent(GeoIpResolverEngine.java:128) [graylog.jar:?]
    at org.graylog.plugins.map.geoip.GeoIpResolverEngine.filter(GeoIpResolverEngine.java:102) [graylog.jar:?]
    at org.graylog.plugins.map.geoip.processor.GeoIpProcessor.process(GeoIpProcessor.java:94) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.handleMessage(ProcessBufferProcessor.java:158) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.dispatchMessage(ProcessBufferProcessor.java:128) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:98) [graylog.jar:?]
    at org.graylog2.shared.buffers.processors.ProcessBufferProcessor.onEvent(ProcessBufferProcessor.java:49) [graylog.jar:?]
    at com.lmax.disruptor.WorkProcessor.run(WorkProcessor.java:143) [graylog.jar:?]
    at com.codahale.metrics.InstrumentedThreadFactory$InstrumentedRunnable.run(InstrumentedThreadFactory.java:66) [graylog.jar:?]
    at java.lang.Thread.run(Thread.java:829) [?:?]

Your Environment

• Graylog Version: 4.3.3+86369d3 (4.3 and earlier)

[drewmiranda-gl]

More detailed steps to reproduce in a lab:

Prereqs: mmdb.zip (link redacted)

1. Stop graylog-server service
2. Place mmdbs from the "old" folder in the mmdb.zip above in /etc/graylog/server/
   • GeoLite2-City.mmdb
   • GeoLite2-ASN.mmdb
3. Start graylog-server
4. overwrite these files with the mmdb files in the "new" folder in mmdb.zip above using mv or cp commands
   • Place in /etc/graylog/server/
5. Tail graylog-server server.log, pretty much instantly the errors will appear

Workaround: Deleting the file (via rm) and then doing a cp/mv appears to prevent this from happening.

[boosty]

@Graylog2/secdev Assuming that SecDev could take this, could we get a high priority for this one?

We had recent issues in Cloud related to geo ip, and I would like to prevent more frustration if possible.

[tellistone]

When this has been solved and we have a timeline for when this will be patched in Cloud, can someone please PM me and also put a note confirming such on HS-961472138.

[damianharouff]

I request the same update for HS-1004828525

[ed-graylog]

Hi, we also have another customer experiencing the same issue. Please see HS-999920232

[boosty]

Thanks a lot for working on this! We would like to roll this out to Cloud as quickly as possible.

Can we get a backport to 4.3 please?

/cc @craig-thomas

[kingzacko1]

@boosty @craig-thomas working on it now, we should have a backport PR in by tomorrow!

[boosty]

@kingzacko1 Thanks a lot 🙌