Update code, use Rufus scheduler, load dictionary better, add "iterate_on"

[guyboertje]

• Move file based dictionary support to separate classes
• Move scheduling to use Rufus Scheduler interval and simplify the read write locking.
• Try to read (huge) CSV, JSON and YAML files in a "streaming" way, i.e. to not read the full file as a string then build a massive hash then update the dictionary hash, but to update the dictionary hash as a key value line is read from disk.
• Abstract the config based dictionary into the same API as the file ones.
• Optimise the three modes of query: exact, exact + regex and regex union.
• Add iterate_on with its three modes of single, array of strings and array of objects.

[colinsurprenant]

@guyboertje did a first pass review. Great refactor overall 👍 Have we made any memory leaks testing to make sure reloading/refreshing dictionaries does not leak?

[guyboertje]

No memory leak testing done as yet.

[colinsurprenant]

@guyboertje it shouldn't be too hard to write dictionary reloading stress tests which should reveal quickly if there are any memory leaks. We could create a separate followup issue for this. I am focusing on this because this is a very typical place where such memory leaks happens.

[colinsurprenant]

LGTM

[ccayg-sainsburys]

https://github.com/logstash-plugins/logstash-filter-translate/pull/67/files#diff-bf70891427c2690568e84ec2c794d12dR4 (require "logstash/util/loggable")

Is, I believe, breaking the functionality of this plugin for us - auto tests which were working now broken with no related change and the following error when trying to start the logstash service:

Couldn't find any filter plugin named 'translate'. Are you sure this is correct? Trying to load the translate filter plugin resulted in this error: no such file to load -- logstash/util/loggable

I presume there is now a dependency that I need to load? But I'm not clear what?

I see this is also now: https://github.com/logstash-plugins/logstash-filter-translate/issues/69