search

joekiller / logstash-kafka

Kafka plugin for Logstash
Apache License 2.0
206 stars 63 forks source link

Feature request: Access to kafka keys in input #38

Closed jpodeszwik closed 9 years ago

jpodeszwik commented 10 years ago

In my company we're using kafka keys to store some metadata like timestamp, etc. We need access to kafka keys in logstash.

I can add those changes myself and make pull requests, but first I need to have some kind of "green light".

Adding that will also require changes in jruby-kafka in Consumer class. We should put some kind of objects to queue instead of strings - that may not be backwards compatible. Should I add another option to juruby-kafka library and give client ability to choose if he wants to receive full objects or just string messages?

joekiller commented 10 years ago

I don't think this would be too difficult. I take it you are using a custom producer and not the logstash output.

Just return the whole iterated object here: https://github.com/joekiller/jruby-kafka/blob/master/lib/jruby-kafka/consumer.rb#L31

and then in the queue_event class we add an option to include the key metadata on the event and otherwise just process msg.message I think around here https://github.com/joekiller/jruby-kafka/blob/master/lib/jruby-kafka/consumer.rb#L31

joekiller commented 9 years ago

@jpodeszwik I've implemented this on master. (https://github.com/joekiller/logstash-kafka/blob/master/lib/logstash/inputs/kafka.rb#L156). Take a look.

joekiller commented 9 years ago

https://github.com/joekiller/logstash-kafka/commit/f874eb498bbedfd9594a4a07f0ac14c016c32809

jpodeszwik commented 9 years ago

@joekiller Didn't have time to look deeper, but it seems fine. Thanx!