search

logstash-plugins / logstash-input-http_poller

Create Logstash events by polling HTTP endpoints!
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-http_poller.html
Apache License 2.0
43 stars 65 forks source link

Allow externalization of urls configuration #46

Open ppf2 opened 8 years ago

ppf2 commented 8 years ago

If there are thousands of servers/endpoints to poll from, the LS config can get unwieldy. It will be nice if we provide a way to externalize the urls configuration (eg. to a file), similar to how you can have jmeter read in a list of http endpoints, etc..

jordansissel commented 8 years ago

a way to externalize the urls configuration (eg. to a file)

My comment may sound silly, but hear me out - The logstash config is already a file. So moving the URLs to another file feels like it's not solving anything? Can you show me an example, maybe? This is one of those cases where managing the logstash configuration by hand is not ideal - generating it with a script is possible (especially with config management tools, like chef and puppet). Thoughts?

andrewvc commented 8 years ago

I do think there's room to do this as a file, so I'm inclined to accept a PR here. I'd like to work on one, but I'm not sure when I'll have a chance.

The file could juts be one URL per line. The plugin uses a key per URL, but we could just assign numbered keys in this case.

jordansissel commented 8 years ago

Each url can have multiple properties, so can we only represent GET requests w/ no configurability?

Put another way, it'd be easier for me to discuss my confusion with the proposal if it were more concrete - a specification or example, perhaps?

ppf2 commented 8 years ago

Thanks for the feedback. This one will probably end up being part of a distributed job/clustering feature request. For the large number of endpoints, we can workaround this by using CM tools and having many LS instances running each with a subset of the endpoints. However, if the user splits the list of urls amount multiple LS indexers, if 1 LS goes down, they will not want to lose say 10-25% of the urls to be processed, and will be nice for another LS instance to pick it up, etc.. So maybe we can revisit this in the future as part of LS clustering.