Add Identity Map Codec (please read the notes to reviewers)

[guyboertje]

also has tests

Follows on from this closed PR on core - https://github.com/elastic/logstash/pull/4104

NOTES TO REVIEWERS:

• This pseudo codec is to be used along with the multiline codec in the file input initially.
• When a stream identity/codec is evicted, it is also flushed using the block given to the decode method which closes over the input-filter queue.
• There will always be one or more buffered lines trapped in the multiline codec because we don't flush on close on any buffered codecs. But if the stream/codec is evicted then the trapped lines will be released.

If you have read or commented on the first PR, these are the important differences:

• There is only one ThreadSafe::Hash that hold both the codec and its eviction time.
• EVICT_TIMEOUT is now 1 hour, read this as - if the stream/codec was last seen more than an hour ago it can be evicted from the map. This needs to be long enough to allow all the multilines to be read and short enough to ensure flushing.
• MAX_IDENTITIES is now 20000
• Cleaner thread only runs when the are stream/codecs mapped

Fixes #10

[andrewvc]

So, to clarify, the intended usage is to be able to call encode(event, identity) as an extension to the current Codec API?

This currently looks great to me code-wise btw!

[andrewvc]

@guyboertje I'd like to hop on a zoom with you monday just to make sure I have all the details straight, but it LGTM!

[ph]

Minor comments, code LGTM!

I have tested this with concurrent-ruby 0.9.1 and tests passes

[ph]

@guyboertje thanks for answering my questions, for concurrent-ruby just make an issue for the migration path. I expect both gem thread_safe and concurrent-ruby to concurrently work ;)

[ph]

Lgtm sorry my last comment wasn't clear enough

[elasticsearch-bot]

Merged sucessfully into master!