Open psychonaut opened 8 years ago
This might be related to https://github.com/mozilla-services/heka/issues/1714. Which versions of Heka and Kafka are you using? And how many partitions are there for the "heka-logs" topic?
heka 0.10.0 and kafka 0.9.0.1
Topic:heka-logs PartitionCount:2 ReplicationFactor:2 Configs: Topic: heka-logs Partition: 0 Leader: 1 Replicas: 1,3 Isr: 3,1 Topic: heka-logs Partition: 1 Leader: 3 Replicas: 3,2 Isr: 3,2
Pinging @trink ...
Might be good to summarize our conversation on IRC a few days ago re: Kafka and Heka
Might be good to summarize our conversation on IRC a few days ago re: Kafka and Heka
Yes, please do :)
@nickchappell can you summarize? :+1:
Heka will soon get a newer version of the Lua sandbox that includes a C Kafka library, librdkafka: https://github.com/edenhill/librdkafka
It supports all of the 0.8 and 0.9 features, including the high-level balanced consumer for v0.9 brokers.
I don't know the timing of when the newer sandbox will get rolled in though, or if it will be in a 0.11 release or be held back until 0.12.
@rafrombrc @trink: care to comment/elaborate?
How soon is soon?
With configuration:
lua_decoders/datacenter.lua is simple injection of datacenter value:
When i use two heka instances with this configuration I've got everything duplicated. When I use three instances I've got every message triple times. Configuration is identical on every instance (so group value is identical as well). I assumed that using group will eliminate duplicates.