search

rapid7 / godap

The Data Analysis Pipeline
MIT License
17 stars 10 forks source link

add/change godaps support with geoip/mmdb #6

Open ssikdar1 opened 5 years ago

ssikdar1 commented 5 years ago

Currently under usage:

$ go get github.com/rapid7/godap
$ sudo bash
# mkdir -p /var/lib/geoip && cd /var/lib/geoip && wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz && gunzip GeoLiteCity.dat.gz && mv GeoLiteCity.dat geoip.dat

However doing a curl -I on the url : 

cam-mbp-5971:dap ssikdar$ curl -I http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
HTTP/1.1 404 Not Found

Looking here at https://dev.maxmind.com/geoip/geoip2/geolite2/ looks like this https://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz the file to wget and decompress.

Looking at the contents:

$ ls ~/Downloads/GeoLite2-City_20190409
COPYRIGHT.txt       GeoLite2-City.mmdb  LICENSE.txt     README.txt

godap will probably also need to change to use an mmdb library like this?

https://github.com/oschwald/geoip2-golang

dabdine-r7 commented 5 years ago

The plan for this is:

  1. We will leave geo_ip and geo_ip_org, but these will be deprecated.
  2. We will add geo_ip2 filter(s).
  3. We will add a compatibility filter to recog to transform geo_ip2 filter data to geo_ip fields. This will allow people to update to geoip2 without the hassle of having to manage downstream changes that may depend on the v1 fields.
dabdine-r7 commented 5 years ago

For reference, here's the GeoIP2 City field list: https://godoc.org/github.com/oschwald/geoip2-golang#City

dabdine-r7 commented 5 years ago

PR: https://github.com/rapid7/godap/pull/13