plugin not thread-safe, only synchronous input, does not track past events

[frapex]

I am currently working on a reimplementation of logstash-filter-throttle to suit the needs of my customer. The reason for this is that the current plugin:

• is not thread-safe (*1)
• only supports synchronous input: with multiple (redis) inputs things get async really quick
• does not track past events: events before "Time.now - period" expire instantly

How it works: when an event is received, the key is stored in a key_cache. The key references a timeslot_cache. The event is allocated to a timeslot (created dynamically) based on the timestamp of the event. The counter is incremented. A second event (same key), within the same "period", is allocated to the same timeslot. The counter is incremented.

This way it is possible to process events from the past in an asynchronous manner. I would like to know whether Elastic would be interested in maintaining this new version of the plugin. Please let me know. I hope this thread is picked up soon and it does not remain unseen.

*1) addisonj already created a pull request for this, which is still not merged unfortunately: https://github.com/logstash-plugins/logstash-filter-throttle/pull/1

[IngaFeick]

+1 Especially for the thread safety problem.

[xo4n]

+1

[AGirin]

I am wondering what is the status of this? We are using thread safe throttle.rb with Logstash 1.4.2 and it works great. But running it with 1.5.4 produces some errors and Logstash eventually stops working. This stops us from upgrading our environment to new version since we user throttle a lot with PagerDuty.

[frapex]

Reimplementation of logstash-filter-throttle plugin is pretty much done. During development it has been put to the test with a test set of about 3 million events.

A new configuration item was added to specify the maximum age of a timeslot. Also, the memory control mechanism has been made adaptive (automatically adjusts timeslot max_age to prevent OOM).

We are currently using the plugin in our production environment.

I will fork the plugin, put code changes online, and create a pull request somewhere next week. Still waiting for a sign of life from Elastic. Guess I need more +1...

[IngaFeick]

Awesome! @frapex

[AGirin]

I copied the file to 1.5.4 Logstash and tried it out. Well - it works fine until we hit throttle event. Once we do - Logstash stops sending ANY events. Not sure if this is our configs but they do work as expected in 1.4.2.

[frapex]

I am pretty sure it is your config, perhaps you can share it. However, I am not sure whether this is the right thread to discuss your problem.

[Hikariii]

+1

[cschotke]

+1

[frapex]

Solved, https://github.com/logstash-plugins/logstash-filter-throttle/pull/5 was merged