[BUG] The location field generated by logstash GeoIP filer does not get recognised as GeoJSON type (geo_point)

[lucabusin]

Describe the bug The geoip filter in logstash generates a geopoint.location field, among others, that should be recognised as geo_json field but is instead split into 2 separate fields, geopoint.location.lon and geopoint.location.lat as numbers. This prevents the location to be used in map visualisations.

To Reproduce Steps to reproduce the behavior:

1. Add this to the logstash filter (in my case the ip address is nested inside log_message)

   filter {
   
   ...
   
   geoip {
       source => "[log_message][forwarded_ip_address]"
   }
   }

2. data is successfully uploaded on my OpenSearch service
3. location is not recognised as a geo_point but instead I see 2 separate number fields geopoint.location.lon and geopoint.location.lat despite the template containing the following:

   "geoip": {
         "dynamic": true,
         "properties": {
           "ip": {
             "type": "ip"
           },
           "latitude": {
             "type": "half_float"
           },
           "location": {
             "type": "geo_point"
           },
           "longitude": {
             "type": "half_float"
           }
         }
       },

Expected behavior geopoint.location should not be split into separate fields but recognised as geo_point type.

Host/Environment (please complete the following information):

• OS: AWS OpenSearch as a service
• Version 1.2.0

Additional context According to this page on AWS OpenSearch docs, ecs_compatibility must be set to disabled. This is relevant to the geoip filter and its mentioned it causes the filter to generate different outputs. At this point I wonder if this is the culprit since I cannot try another installation of OpenSearch other than the one I have on AWS.

Here is my "full" logstash config if relevant:

input {
    file {

        ...

        codec => json {
            target => "[log_message]"
        }
  }
}

filter {

    ...

    geoip {
        source => "[log_message][forwarded_ip_address]"
    }
}

output {

  ...

  opensearch {

        ...

        ecs_compatibility => disabled
  }
}

[dblock]

Do you have a ticket with the Managed AWS OpenSearch support yet? I think you should open one.

[lucabusin]

Hi, @dblock Sorry. No I haven't. I'll do that. So you are saying that what I've done looks ok from what you can see?

[dblock]

Am I wrong to read this as something specific to the Amazon Managed OpenSearch service? Can you reproduce the same problem in OpenSearch OSS?

[MrakoMaks]

I have a similar problem.

Opensearch 2.3.0 Logstash OSS 8.4.0 Ubuntu 20.04.5

filter:

geoip {
        database => "/opt/logstash/geoip/GeoLite2-City.mmdb"
        source => "visitor_ip"
        target => "geoip"
}

In discover, I see only geoip.geo.location.lon and geoip.geo.location.lat

At the same time, there is data in json:

"geoip": {
      "geo": {
        "location": {
          "lon": 24.7323,
          "lat": 59.433
        },
        "country_name": "Estonia",
        "country_iso_code": "EE",
        "city_name": "Tallinn"
      },
      "ip": "XX.XX.XX.XX"
    }
  },

[MrakoMaks]

I have a similar problem.

Opensearch 2.3.0 Logstash OSS 8.4.0 Ubuntu 20.04.5

filter:

geoip {
        database => "/opt/logstash/geoip/GeoLite2-City.mmdb"
        source => "visitor_ip"
        target => "geoip"
}

In discover, I see only geoip.geo.location.lon and geoip.geo.location.lat

At the same time, there is data in json:

"geoip": {
      "geo": {
        "location": {
          "lon": 24.7323,
          "lat": 59.433
        },
        "country_name": "Estonia",
        "country_iso_code": "EE",
        "city_name": "Tallinn"
      },
      "ip": "XX.XX.XX.XX"
    }
  },

The problem was solved by adding a parameter ecs_compatibility => "disabled"