search

logstash-plugins / logstash-filter-translate

Translate filter for Logstash
Apache License 2.0
21 stars 47 forks source link

Feature/redis backend support #41

Closed bananabr closed 7 years ago

bananabr commented 7 years ago

I really would like to see this on the main stream. This is a first working version with lots of room for improvements, but being able to use REDIS as a dictionary backend has a lot of benefits. In my company for example, we wrote an agent that listens to Active Directory network logon Events and record an "ip => user" entry on a REDIS database. This way we can use the translate filter to map every firewall, ssh and other logs that contain an IP address to a user at an especific point in time.

jordansissel commented 7 years ago

It feels like this belongs as a separate filter, not necessarily in the translate filter?

The reason I say this is because Redis may have many different configuration options (credentials, sentinel or not, multiple hosts, different key scenarios, etc). These configurations would be specific to Redis usage.

If we extend this idea further, and consider other backends, each backend seems like it's going to have unique settings. For that reason, I think this feature belongs as a separate plugin.

jordansissel commented 7 years ago

ALso, thanks for working on this feature :)

bananabr commented 7 years ago

@jordansissel I thought about it before starting to work no it, but the it "feels" right to think that the translate filter should be the one used to translate fields regardless of the dictionary backend. I starting to work today on BerkleyDB support, as we have another use case that demans it. Couldn't we use an idea similar to codecs? Every "dictionary backend" would have an associated hash property "backend options" which would contain all the backend specific options. The backend would be identified by the protocol prefix in dictionary_path "redis://, file:// ...". For compatibility puporses, local file backend would be the default and would not require a protocol prefix.

bananabr commented 7 years ago

If you still think is better for this to be a separate filter, no problem, but in this case, what do I need to do for it to available remotely to the "plugin install command"?

guyboertje commented 7 years ago

@bananabr There is a community effort to redesign the remote dictionary kind of lookup and enhancement. Here is the first PR for the initial code https://github.com/logstash-plugins/logstash-filter-lookup/pull/1. Perhaps you can work with the folks on this plugin?

guyboertje commented 7 years ago

@bananabr I like your idea of a modular approach to backends. Perhaps you could influence the code towards that direction in the lookup filter.

guyboertje commented 7 years ago

@bananabr

If you still think is better for this to be a separate filter, no problem, but in this case, what do I need to do for it to available remotely to the "plugin install command"?

We download plugins from rubygems.

You need to create an account on rubygems.org, there are some guides there about how to 'publish' your modified filter. The gemspec has metadata so we only install gems that are "logstash_plugin" => "true"

bananabr commented 7 years ago

@guyboertje thanks for the directions you gave me. I guess I will probably publish my own filter at first and work on a merge attempt with the lookup plugin in the weekend.