search

reiseburo / hermann

A gem providing cross-platform Kafka producer and consumer support
MIT License
72 stars 76 forks source link

Ability to configure Hermann #39

Open StabbyCutyou opened 10 years ago

StabbyCutyou commented 10 years ago

Currently, there is very little client side configuration for Hermann, and by extension librdkafka.

For example, I'm running into an issue where I'd love to edit 'queue.buffering.max.messages', but lack the ability. This is causing issues where I need to introduce artificial latency into my work to get around messages being dropped from the local queue (effectively, the same issue as resolved here: https://github.com/edenhill/librdkafka/issues/117)

What would be great would be an options hash accepted into the Hermann producer / consumer class constructors, such that Hermann could intercept Hermann specific values, and transform/pass on librdkafka specific values.

For example, you could pass an options hash like

opts = { "queue.buffering.max.messages" => 100000 }

p = Hermann::Producer.new(blah,blah,opts)

and pass it directly to librdkafka - This has the added benefit of making the config options for Hermann code 1:1 with librdkafka (albeit less ruby-ish).

trane commented 10 years ago

The way that jruby-kafka does this is to just create a list of known options and check those that you specify against it, then send the validated options to the kafka producer: https://github.com/joekiller/jruby-kafka/blob/master/lib/jruby-kafka/producer.rb

Perhaps a Hermann::Producer::Config that abstracts over both librdkafka and apache kafka?

StabbyCutyou commented 10 years ago

IMHO the actual implementation doesn't matter as much, but I'd say the JRuby client leans more towards "java" than "ruby" with that implementation - A lot of ruby developers are more akin to having options be the final, optional member of a signature.

The real important piece is just being able to do configuration. Currently, this is preventing me from using this gem in my benchmarks because the defaults can't keep up with the scale I need to benchmark it at.

trane commented 10 years ago

Sure, I wasn't suggesting to use the jruby-kafka method. I was saying that since we have multiple ways to configure the potential backends (librdkafka, kafka), it would make sense to abstract over the backend with a Hermann::Producer::Config that is not implementation specific.

StabbyCutyou commented 10 years ago

Sure, that makes sense.