WIP: #6 Support GeoIP2 lookups

[dabdine-r7]

What is this?

This is a work in progress patch to support the Maxmind GeoIP2 database.

This patch introduces two new filters: geo_ip2_city and geo_ip2_isp, which replace the now deprecated geo_ip and geo_ip_org filters, respectively.

Geoip files have now been re-organized to allow for reuse of common features:

1. geoip.go / geoip_test.go - Code and tests which which contain the "API" definitions for supporting GeoIP lookups (namely, GeoIPDecoder, FilterGeoIP and the New method)
2. geoip_legacy.go / geoip_legacy_test.go - Code and tests for geo_ip and geo_ip_org
3. geoip2.go / geoip2_test.go - Code and tests for geo_ip2_city and geo_ip2_isp. Since GeoIP2 supports multiple languages, you can specify which language you want using the GEOIP2_LANGUAGE environment variable (defaults to "en", see https://support.maxmind.com/geoip-faq/geoip2-and-geoip-legacy-implementation/what-languages-does-geoip2-support/). The path to the database can be specified with GEOIP2_CITY_DATABASE_PATH for geo_ip2_city, or GEOIP2_ISP_DATABASE_PATH for geo_ip2_isp, respectively.

Example output:

echo 81.2.69.142 | GEOIP2_LANGUAGE=en GEOIP2_CITY_DATABASE_PATH=test/test_data/geoip2/GeoIP2-City-Test.mmdb godap lines + geo_ip2_city line + json | jq .
{
  "line": "81.2.69.142",
  "line.geoip2.city.geoname_id": 2643743,
  "line.geoip2.city.name": "London",
  "line.geoip2.continent.code": "EU",
  "line.geoip2.continent.geoname_id": 6255148,
  "line.geoip2.continent.name": "Europe",
  "line.geoip2.country.geoname_id": 2635167,
  "line.geoip2.country.is_eu": true,
  "line.geoip2.country.iso_code": "GB",
  "line.geoip2.country.name": "United Kingdom",
  "line.geoip2.location.accuracy_raidus": 10,
  "line.geoip2.location.latitude": 51.5142,
  "line.geoip2.location.longitude": -0.0931,
  "line.geoip2.location.metro_code": 0,
  "line.geoip2.location.time_zone": "Europe/London",
  "line.geoip2.postal.code": "",
  "line.geoip2.registered_country.geoname_id": 6252001,
  "line.geoip2.registered_country.is_eu": false,
  "line.geoip2.registered_country.iso_code": "US",
  "line.geoip2.registered_country.name": "United States",
  "line.geoip2.represented_country.geoname_id": 0,
  "line.geoip2.represented_country.is_eu": false,
  "line.geoip2.represented_country.iso_code": "",
  "line.geoip2.represented_country.name": "",
  "line.geoip2.represented_country.type": "",
  "line.geoip2.subdivisions.0.geoname_id": 6269131,
  "line.geoip2.subdivisions.0.iso_code": "ENG",
  "line.geoip2.subdivisions.0.name": "England",
  "line.geoip2.subdivisions.length": 1,
  "line.geoip2.traits.is_anon_proxy": false,
  "line.geoip2.traits.is_satellite": false
}

$ echo 1.128.0.0 | GEOIP2_LANGUAGE=en GEOIP2_ISP_DATABASE_PATH=test/test_data/geoip2/GeoIP2-ISP-Test.mmdb godap lines + geo_ip2_isp line + json | jq .
{
  "line": "1.128.0.0",
  "line.geoip2.isp.asn": 1221,
  "line.geoip2.isp.asn_org": "Telstra Pty Ltd",
  "line.geoip2.isp.isp": "Telstra Internet",
  "line.geoip2.isp.org": "Telstra Internet"
}

An example of the geo_ip2_legacy_compat filter:

$ echo 8.8.8.8  | GEOIP2_CITY_DATABASE_PATH=~/Downloads/GeoLite2-City_20190416/GeoLite2-City.mmdb ./godap lines + geo_ip2_city line + geo_ip2_legacy_compat line + remove_prefix line.geoip2. + json | jq .  
{
  "line": "8.8.8.8",
  "line.country_code": "US",
  "line.country_name": "United States",
  "line.latitude": 37.751,
  "line.longitude": -97.822
}

TODO

[ ] Add more tests (integration, unit) [ ] Update documentation [x] Add backward compatibility filter for transforming geoip2 fields back to the geoip legacy field format (likely to be named geo_ip2_legacy_compat)

[dabdine-r7]

Technically speaking, for he compat filter, we should set the country code to A1 or A2 based on the values of the geoip2 traits object. The geoip2 what's new link details this:

https://dev.maxmind.com/geoip/geoip2/whats-new-in-geoip2/

[dabdine]

@tsellers-r7 any chance we can get this merged :)

[dabdine]

Still needs documentation, likely needs to be tested again against the DAP bats tests jhart wrote.

[tsellers-r7]

Hurm, I thought we rm -rf this whole repo.. 😈 I'll try to take a look soonish. Lemme know if there is any priority attached to it.