search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.11k stars 3.48k forks source link

Provide a general purpose API for extended plugin configuration #8582

Open andrewvc opened 6 years ago

andrewvc commented 6 years ago

Currently plugins like the grok filter use files for extended configuration options like https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-patterns_dir to handle some options. These things are hard to configure centrally however. Ideally there'd be a way to centralize these options, perhaps by letting them be satisfied by an arbitrary URL, not just a central file.

jordansissel commented 6 years ago

@andrewvc let's get a list of plugins which use files for extra configuration and see if any obvious abstractions come of it.

letting them be satisfied by an arbitrary URL

The impetus for this is grok patterns being hard to manage when central config is used. I agree with the problem statement. A URL alone is not sufficient because the "obvious" (to me) place to store patterns is Elasticsearch (when central config is enabled), and Elasticsearch does not serve plain text responses outside of the /_cat api.

andrewvc commented 6 years ago

@jordansissel good point! I'm +1 on a more thorough investigation of real world plugin usage first.

WRT Elasticsearch, ES does have URLs, we could have some provision for pulling special JSON fields etc., but I agree that's a question to ask later.