Add a simulate endpoint for Logstash

[suyograo]

Add a simulate endpoint as a convenience to test Logstash's filters in a pipeline. This is inspired from ingest's simulate API which is super useful. Today, if someone wants to test filters in dev/staging, they'd have to add stdin and stdout which means changing the existing pipeline config. This API can make it easy to test iterations to their config.

Behavior

1. Using a named pipeline which has already been loaded from pipeline.yml.

Note: If the named pipeline has inputs/outputs, we ignore that and don't use them. This operation should have no side-effects to the original pipeline.

Request

POST localhost:9600/_node/pipeline/<pipeline_name>/_simulate
{
   "events": [
      "183.60.215.50 - - [11/Sep/2014:22:00:00 +0000] \"GET /scripts/netcat-webserver HTTP/1.1\" 200 182 \"-\" \"Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)"
   ]
}

Response is JSON formatted output of the Event(s) after running it through the filters.

{
   "events": [
    {
        "host": "node1",
        "blah": "foo"
     }
]

---

2. Using a pipeline definition provided via the body:

POST localhost:9600/_node/pipeline/_simulate 
{
     // pipeline definition as a string
    "pipeline_config" : "filters { geoip { source => \"clientIP\" } }"
    "events": [
      "183.60.215.50 - - [11/Sep/2014:22:00:00 +0000] \"GET /scripts/netcat-webserver HTTP/1.1\" 200 182 \"-\" \"Mozilla/5.0 (compatible; EasouSpider; +http://www.easou.com/search/spider.html)"
   ]
}

Response is JSON formatted output of the Event(s) after running it through the filters.

{
   "events": [
    {
        "foo": "bar",
       "blah": "foo"
     }
]

[suyograo]

We could then use this from config management UI.

[jordansissel]

This provides a significant attack surface because we do not have a security mechanism for the API.

[jsvd]

++ at best I'd see this only working if the puma server was bound to localhost. If a user goes to the trouble of redirecting the tcp traffic from localhost to an outside interface we can't prevent it, but that's ok.

WRT to config management, this restriction could be lifted once x-pack is installed and x-pack is enabled.

[jordansissel]

@jsvd x-pack doesn't provide protection for Logstash's HTTP API. It could, but does not right now.

[jsvd]

x-pack doesn't provide protection for Logstash's HTTP API. It could, but does not right now.

Sorry, I meant to imply that that would have to be implemented as pre-requisite.

[suyograo]

I understand the point about security, but I'd like to find a middle ground here to provide this functionality to users.

How about we add functionality which allows users to simulate only existing pipelines. Pipelines defined in yml only. Basically, behavior #1 described above. That way, users will not be able to send arbitrary configuration. They can only send data to simulate existing pipelines.