Geoip and useragent filters not work

[pacolotero]

Always returns the same result and coords ("Ohher" and "0.0")

            "ua" => {
           "name" => "Other",
             "os" => "Other",
        "os_name" => "Other",
         "device" => "Other"
    },
         "geoip" => {},
        "coords" => [
       [0] 0.0,
       [1] 0.0
    ]
}

This is my logstash.conf

filter {
  mutate {
  convert => [ "responsecode", "integer",
  "responsesize", "integer", "resp_time", "integer",
  "geoip.latitude", "float", "geoip.longitude", "float" ]
  }
  useragent {
    source => "user_agent"
    target => "ua"
  }
  geoip {
    source => "sourceip"
  }
  mutate {
  add_field => [ "coords", "%{[geoip][longitude]}",
               "tmplat", "%{[geoip][latitude]}" ]
  }
  mutate {
  merge => [ "coords", "tmplat" ]
  }
  mutate {
  convert => [ "coords", "float" ]
  remove => [ "tmplat" ]
  }
}

[jordansissel]

The 'tmplat' stuff you're using seems to have come from an old blog post or something; creating the 'coords' field is now done by default as [geoip][coordinates] is provided for you.

However, specific to your problem, can you add 'output { stdout { codec => rubydebug } }' to your config and show the full event?

[pacolotero]

The new logstash.conf

input {
  file {
    type => "rails_log"
    path => "/home/user/rails/log/logstash_production.log"
    codec =>   json {
      charset => "UTF-8"
    }
  }
}

filter {
  useragent {
    source => "useragent"
  }
  geoip {
    source => "remote_addr"
    add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
    add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}" ]
  }
  mutate {
    convert => [ "[geoip][coordinates]", "float" ]
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    host => localhost
    workers => 3
  }
}

The output of full event

{
       "@source" => "unknown",
         "@tags" => [
        [0] "request"
    ],
       "@fields" => {
            "method" => "GET",
              "path" => "/",
            "format" => "html",
        "controller" => "home",
            "action" => "index",
            "status" => 200,
          "duration" => 9.07,
              "view" => 8.5,
                "ip" => "83.44.94.62",
             "route" => "home#index",
        "request_id" => "e88f59a7f2b60213c4e689263e98d080",
        "parameters" => {}
    },
    "@timestamp" => "2014-10-21T17:44:07.610+02:00",
      "@version" => "1",
          "type" => "rails_log",
          "host" => "galactus",
          "path" => "/home/user/rails/log/logstash_production.log",
          "name" => "Other",
            "os" => "Other",
       "os_name" => "Other",
        "device" => "Other",
         "geoip" => {}
}

[jordansissel]

your useragent filter is looking for a field named 'useragent' and you have no such field.

your geoip is looking for a field named 'remote_addr' and you have no such field, but you do have one accessible under @fields called 'ip', so maybe source => "[@fields][ip]" is what you want?

Your log format looks like the older version of the logstash json schema (you have @source, @tags, and @fields and your @timestamp is in the wrong timezone. Not sure what library you're using to emit these logs, but it seems a bit old :P

[pacolotero]

Thanks! works perfectly.

I use logstasher to convert my rails log (https://github.com/shadabahmed/logstasher)

[timbunce]

Looks like this issue could be closed now.