Fields not being sent in parameters

[jarlrmai]

Plugin is sending word "postcode" as a parameter rather than returned field when using below with logstash 2.3.4

input {
  # Read all documents from Elasticsearch matching the given query
  elasticsearch {
    hosts => "localhost"
    index => "postcodes"
    scan => false
    size => 1
    scroll => "1m"
    #query => '{"from":0,"size":10,"query":{"match_all":{}}}'
    query => '{"query":{"range":{"id":{"gte":1,"lte":2}}}}'
    }
}

filter {
    rest {
      request => {
        url => "https://maps.googleapis.com/maps/api/distancematrix/json?"
        json => false
        method => "get"
        sprintf => true
        params => {
          "units" => "imperial"
          "origins" => "%{postcode}"
          "destinations" => "A119ZZ"
          "key" => "GOOGLEAPIKEY"
        }
      }
    }
}
output {
        stdout { codec => rubydebug }
        }

[gandalfb]

Hi,

if I see this, the "Set to Rest Filter Get with params sprintf" testcase should handle this.

Could you please add information about the event and plugin version?

[gandalfb]

I guess this is caused as your sprintf key is used within request {} not rest {} Please check usage as shown in README

Thanks

[jarlrmai]

Thanks I amended filter to be as you suggested now it sends blank values am I still doing something wrong?

input {
  # Read all documents from Elasticsearch matching the given query
  elasticsearch {
    hosts => "localhost"
    index => "postcodes"
    scan => false
    size => 1
    scroll => "1m"
    #query => '{"from":0,"size":10,"query":{"match_all":{}}}'
    query => '{"query":{"range":{"id":{"gte":1,"lte":2}}}}'
    }
}

filter {
    rest {

      request => {
        url => "https://maps.googleapis.com/maps/api/distancematrix/json?"
        method => "get"

      params => {
        "units" => "imperial"
        "origins" => "%{postcode}"
        "destinations" => "SW1A1AA"
        "key" => "APIKEY"
        }
        }
      json => true
      sprintf => true
      target => "google_response"
    }

}
output {
        stdout { codec => rubydebug }
        }

[jarlrmai]

Below is the output from rubydebug

Settings: Default pipeline workers: 4
Pipeline main started
{
           "postcode" => "BB01GR",
                 "id" => 1,
           "@version" => "1",
         "@timestamp" => "2016-09-13T12:14:44.032Z",
    "google_response" => {
        "destination_addresses" => [
            [0] "London SW1A 1AA, UK"
        ],
             "origin_addresses" => [
            [0] ""
        ],
                         "rows" => [
            [0] {
                "elements" => [
                    [0] {
                        "status" => "NOT_FOUND"
                    }
                ]
            }
        ],
                       "status" => "OK"
    }
}

[gandalfb]

Hi, thanks for further information.

I tried your example and some trivial google example within chrome. Somehow it looks like your origins is not processed by google properly; nothing related to the plugin, I guess. :smile: Your copied document is a "perfect" representation of the google api, which is also shown to me via chrome.

https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=BB01GR&destinations=SW1A1AA&key=[key]

and

https://maps.googleapis.com/maps/api/distancematrix/json?units=imperial&origins=Washington,DC&destinations=New+York+City,NY&key=[key]

both the browser and rest plugin show the same response. You could try to verify with the config:

        filter {
          rest {
            request => {
              url => "https://maps.googleapis.com/maps/api/distancematrix/json?"
              method => "get"
              params => {
                "units" => "imperial"
                "origins" => "%{postcode}"
                "destinations" => "New York City,NY"
                "key" => "KEY"
              }
            }
            json => true
            sprintf => true
            target => "google_response"
          }
        }

Whereas the postcode field should be "Washington,DC"

This leads to the response:

       {
          "destination_addresses" : [ "New York, NY, USA" ],
          "origin_addresses" : [ "Washington, DC, USA" ],
          "rows" : [
             {
                "elements" : [
                   {
                      "distance" : {
                         "text" : "225 mi",
                         "value" : 361713
                      },
                      "duration" : {
                         "text" : "3 hours 48 mins",
                         "value" : 13658
                      },
                      "status" : "OK"
                   }
                ]
             }
          ],
          "status" : "OK"
       }

Or check within the browser, which origins and destinations returns a result you know is true and check with a corresponding elastic document.

Hope this helps to narrow it down further.

[jarlrmai]

Yeah you are right google doesn't like the postcodes! I tried it with known ones and all okay

Thanks! This plugin is really useful and will help us loads.