reiseburo / hermann

A gem providing cross-platform Kafka producer and consumer support
MIT License
71 stars 76 forks source link

Update producer default settings #112

Open jamescway opened 9 years ago

jamescway commented 9 years ago
#Current defaults
      DEFAULTS = {
                    'partitioner.class'     => 'kafka.producer.DefaultPartitioner',
                    'request.required.acks' => '1',
                    'message.send.max.retries' => '0'
                  }.freeze

I'm thinking we should make: request.required.acks = -1 (needs ack from all ISR's, best durability) message.send.max = 3 (to made the tradeoff towards reliability, when the cost is possibility of dups)

rtyler commented 9 years ago

I spoke briefly with @polynomial about this and there is definitely some trade-off to be made here. Before we change the defaults, we should probably do a lot more experimentation to take a simulated load and execute it against a Kafka cluster with/without the minimum required acks.

There's also some questions I have around the use/exposure of the min.insync.replicas setting as it relates to Hermann.