http keep alive

[gandalfb]

I had the behaviour of the rest_client not using http-keep-alive. Leading to a really inefficient connection usage and poor performance.

On the http_poller plugin I stumbled upon the mixing HTTPClient, which supports this.

So I made some changes on the config section, with - maybe - including some "logstash" style documentation. Validation and request setup mostly is sourced from the http_poller plugin, but omitting the url regex check, to support field references.

Further changes:

• response_keyrenamed to target which is used by other logstash plugins as well
• error handling via event tags, not fields
  • json or rest errors can be set to custom tags (array)
• fallback functionality to add a default hash, in case of http error (http code not 200 to 299) or json error
• request represents the HTTPClient config
• it is focussed on json
  • if it is post, the paramsare saved as bodyand dumped as json
  • for json the Logstash::JSON is used instead of "require json"
  • string only functionality is not well evolved
• extended test cases with travis-ci usage
• paramsmay also be used for a get
• url, body, params are sprintf'ed
• remove git client dependency within .gemspec; this lead to errors once installing the plugin on a remote server without git

First "live" tests in an integration env show quite a performance boost and a really small number of tcp connections (1 or 2 reused connections instead of ~400 waiting connections), which solves my initial issue including a fallback handling. Further parameters are also configurable within the Manticore object (request) => logstash-mixin-http_client

Maybe this is something useful, although the config syntax is changed, implicating a major version?

[lucashenning]

Awesome work @gandalfb! I appreciate your contribution. Guess it's a good approach to align the style to the logstash standards and I like the performance improvement.

You're right, this should be a major version. However, I guess we are still away from a stable version. So, if you don't mind I'll call this one v0.2.0.

[lucashenning]

Just published the new gem v0.2.0: https://rubygems.org/gems/logstash-filter-rest/versions/0.2.0 Thanks again for your help. Hope I'll find time to keep working on this...