Allow grok pattern to be passed in as a parameter

[Furb13]

I have a complex use case where I want to store meta data about an event being parsed by logstash in a dictionary. In there, some fields may need additional grok on one of their fields already parsed out by kv plugin. This is why I need to pass in the grok pattern from an existing filed (variable).

Discussion pertaining to this: https://discuss.elastic.co/t/how-to-get-grok-filter-from-variable/118511

[jordansissel]

I think I understand what you want (the grok pattern for a given event to be sourced from a field in that event). However, I'm not sure I understand what exactly you are trying to achieve?

It seems like you have a translate filter intended to provide a grok pattern for later use, but can you explain more about what you have in the translate dictionary? How many patterns? Are they all unique?

From the looks of things, please correct me if I am wrong:

• You have the translate filter look up event_id and load the intended grok pattern.
• Use that grok pattern in the next grok as the contents of the ec_grok field

right? If so, other cases users are using if statements, often generated programmatically:

if [event_id] == "foo" {
  grok { ... } # the `foo` grok
} else if [event_id] == "bar" {
  grok { ... } # the `bar` grok
} else if ...

Is what you are doing an attempt to write this same mechanism but using translate instead?

[Furb13]

That is correct. As you can image in the case of a firewall, there would be grok patterns for some of the events (your probably looking at 25-50 patterns) is would be easier in my mind to store them in a dictionary instead and get them through the translate method. This would prevent us from having a massive logstash.conf file. The challenge I currently face is that unlike other plugins there is no way to reference %{ec_grok} as a variable. I get this may be by design, but for complex use cases I can see this being extremely helpful.

[krzysztof-kalinowski]

I agree, this could be very usefull feature :)