search

logstash-plugins / logstash-input-http_poller

Create Logstash events by polling HTTP endpoints!
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html
Apache License 2.0
43 stars 65 forks source link

Target option on the config #13

Closed elvarb closed 9 years ago

elvarb commented 9 years ago

If you poll a json api you get back a json object.

In some cases it would be helpful to have a target option in the config to dictate under what object the json object should fall into.

For example if you are using the http poller to query multiple system types so each object would fall to the correct place. This would minimize the risk of mapping issues on the elasticsearch end if you have two api's and one gives the field error=500 while the other is error=ThingBroken.

maximede commented 9 years ago

Wouldn't you use two different input type and save it to two different indexes in this case ?

andrewvc commented 9 years ago

@elvarb is there something here that couldn't be accomplished via a mutate filter? I'd rather not try to solve stuff here that could be solved via a filter.

elvarb commented 9 years ago

As an input only plugin by adding a tag to the input and use a condition for that tag to mutate all the values. How would you use the mutate filter for this use case?

The json filter has a target option, https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html

So the solution here would then to skip having http_poller convert the message to a json object and then use the json filter to convert the raw text to a json object under a target name.

andrewvc commented 9 years ago

@elvarb I think that makes a lot of sense re: the target option, I'll try to get that in the initial gem release.

andrewvc commented 9 years ago

I'm realizing that one shortcoming of a target option is that codecs inject @ metadata. It's really not ideal in terms of unnesting the root data to a child key. I think we should have this feature, but I think I may also submit an issue to logstash core.

andrewvc commented 9 years ago

@elvarb I'm going to implement this today, but there is one caveat that will require a Logstash core change in the future: https://github.com/elastic/logstash/issues/3607