Add timeout option to grok filter

[ppf2]

We have users sending in occasional large messages of many Mbs, and if they have not implemented logic in the pipeline to filter out these messages, the grok filter will run and peg the CPU threads for a very long time. It will be helpful to have a default (or configurable) timeout for grok filters so that it will give up at some point, etc..

[jordansissel]

We used to have a watchdog that caught runaway filters in these situations. We can bring it back - good idea :)

On Thursday, December 10, 2015, Pius notifications@github.com wrote:

We have users sending in occasional large messages of many Mbs, and if they have not implemented logic in the pipeline to filter out these messages, the grok filter will run and peg the CPU threads for a very long time. It will be helpful to have a default (or configurable) timeout for grok filters so that it will give up at some point, etc..

— Reply to this email directly or view it on GitHub https://github.com/logstash-plugins/logstash-filter-grok/issues/65.

[jordansissel]

Related: Joni (as of 2.5-ish years ago) supports being interrupted via Thread.interrupt.

[jordansissel]

Proposal for this:

1) Configurable execution timeout. Default 0.5 seconds. 2) When execution time expires, we interrupt grok via java Thread.interrupt(). 3) On interruption, we tag the event _grokexecutiontimeout and continue on to the next filter in the filter worker. 4) On interruption, we log a warning indicating that Grok's execution expired and include the following context: the text input, the original pattern being tested, and the execution time.

This may get tricky given the new complexity of the combined filter-and-output worker units as well as having some possibility (we'll need to protect against) of race conditions of grok succeeding vs the timer expiring.

It would be more-ideal to use cpu-time instead of wall clock time to avoid external resource contention for cpu or scheduling causing us to mistakenly trigger an execution timeout, but thread-specific cpu usage is not easily available on any OS (as far as I can tell...)

[purbon]

I think your last comment will have a very nice improvement for grok, specially when problematic situation happen.

+1 for them.

[wiibaa]

@suyograo can we close this as duplicate of #82 ?

[magicdude4eva]

Perhaps related - I have no idea why a simple regex times out - https://github.com/logstash-plugins/logstash-filter-grok/issues/99

[jsvd]

this was implemented in https://github.com/logstash-plugins/logstash-filter-grok/pull/88