elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.18k stars 3.5k forks source link

Add more MaxMind GeoIP data sets - Cloud and ECE benefit? #10108

Open hookhax opened 5 years ago

hookhax commented 5 years ago

MaxMind offers expanded data sets for purchase. We want to start using those. I see that documentation exists for having the geoip plugin use an alternative database. These databases update frequently, however, even daily. That means we need a process to keep the plugin updated, make sure updates are atomic (no removing old geoip database until new one is validated), and babysit generally (did it run? were there errors?). Headaches.

Can Cloud and ECE provide this instead? Practically, it makes sense for there to be one centrally managed super-geoip database instead of clients all rolling their own and maintaining it.

Data points needed: is_anonymous is_anonymous_vpn is_hosting_provider is_public_proxy is_tor_exit_node network isp organization autonomous_system_number autonomous_system_organization connection_type

Databases listed here: https://dev.maxmind.com/geoip/geoip2/geoip2-anonymous-ip-csv-database/ https://dev.maxmind.com/geoip/geoip2/geoip2-isp-csv-database/ https://dev.maxmind.com/geoip/geoip2/geoip2-connection-type-csv-database/

Thanks for considering.

m-cameron commented 5 years ago

Those are commercial database. Why not supporting free IP2Proxy LITE proxy database from https://lite.ip2location.com/database/px4-ip-proxytype-country-region-city-isp

You can detect VPN, TOR, open proxy, web proxy and data center range using it.

hookhax commented 5 years ago

That's a good source too, albeit with fewer data points. I mean to outsource the setup and process management effort to Elastic though. The true cost is not in paying for commercial data but in management/troubleshooting of the update process.

I am going to try your link on a personal instance. Thanks!