logstash-plugins / logstash-filter-geoip

Apache License 2.0
64 stars 80 forks source link

netflow codec geoip lookup not working #144

Open sgreszcz opened 5 years ago

sgreszcz commented 5 years ago
# Refer to this: https://www.elastic.co/guide/en/logstash/current/plugins-codecs-netflow.html
input{
    udp{
        port => 2055
        codec => netflow
        tags => ["netflow"]
        id => "netflow_input"
    }
}

filter{

# Enhance with ISP geolocation information using free GeoIP2 ISP Database

    geoip {
        database => "/usr/share/logstash/geolite2/test.mmdb"
        source => "ipv4_src_addr"
    }
}

output {
    if "netflow" in [tags]{
        elasticsearch{
        hosts => "${ELASTICSEARCH_HOST}:${ELASTICSEARCH_PORT}"
        index => "netflow-%{+YYYY.MM.dd}"
        id => "netflow_es_output"
    }
    #stdout{ }
    }
}

Also tried:

filter{
    geoip {
        database => "/usr/share/logstash/geolite2/test.mmdb"
        source => "netflow.ipv4_src_addr"
    }
}

The test.mmdb is a MaxMind DB with internal private IP geo mappings.

Using Logstash Netflow codec for example, and trying to source off of "ipv4_src_addr" or "netflow.ipv4_src_addr":

{
  "_index": "netflow-2018.11.13",
  "_type": "doc",
  "_id": "q4thD2cBb3n6NizXdq3b",
  "_version": 1,
  "_score": null,
  "_source": {
    "netflow": {
      "l4_dst_port": 56335,
      "src_tos": 0,
      "ipv4_src_addr": "10.230.91.4",
      "input_snmp": 7,
      "flow_sampler_id": 0,
      "protocol": 17,
      "src_mask": 25,
      "in_pkts": 80,
      "tcp_flags": 0,
      "version": 9,
      "ipv4_dst_addr": "173.38.200.85",
      "l4_src_port": 161,
      "flow_seq_num": 208343981,
      "first_switched": "2018-11-13T23:21:41.999Z",
      "dst_as": 0,
      "ipv4_next_hop": "10.230.2.98",
      "flowset_id": 257,
      "dst_mask": 21,
      "output_snmp": 8,
      "last_switched": "2018-11-13T23:22:08.999Z",
      "in_bytes": 68403,
      "src_as": 0
    },
    "tags": [
      "netflow",
      "_geoip_lookup_failure"
    ],
    "@version": "1",
    "host": "172.19.0.1",
    "@timestamp": "2018-11-13T23:22:23.000Z"
  },
  "fields": {
    "netflow.first_switched": [
      "2018-11-13T23:21:41.999Z"
    ],
    "@timestamp": [
      "2018-11-13T23:22:23.000Z"
    ],
    "netflow.last_switched": [
      "2018-11-13T23:22:08.999Z"
    ]
  },
  "sort": [
    1542151343000
  ]
}

Debugs always tag output with _geoip_lookup_failure

The DB is valid and works OK with the python geoip2 library:

import geoip2.database
reader = geoip2.database.Reader('/Users/me/Downloads/test.mmdb')
response = reader.city('10.50.172.174')

response.location.latitude
51.445324
response.location.longitude
-0.443962

response
geoip2.models.City({'city': {'names': {'en': ' FELTHAM'}}, 'continent': {'code': 'something'}, 'country': {'iso_code': ' UNITED KINGDOM', 'names': {'en': ' UNITED KINGDOM'}}, 'location': {'accuracy_radius': 1000, 'latitude': 51.445324, 'longitude': -0.443962}, 'traits': {'ip_address': '10.50.172.174'}}, ['en'])
hkelley commented 3 years ago

Any updates here? I have the same scenario. We have inserted our RFC1918 network into the GeoLite2-City.mmdb DB and would like to use these with geoip .