logstash-plugins / logstash-filter-translate

Translate filter for Logstash
Apache License 2.0
21 stars 47 forks source link

Switch to async and swap loading of the dictionary. #65

Closed guyboertje closed 6 years ago

guyboertje commented 6 years ago

Currently the loading of dictionary off disk is done by stealing some time from the filter call so large dictionary loads will impact throughput especially if dictionary reloading is done quite often. If this is done on a separate thread outside of any locking and then only lock to swap the reference. The filter call will only wait on the lock if the swap is taking place as the filter call is initiated.

The downside is that, until GC reclaims the old dictionary, there will be two copies of each file in memory.