search

kafkaex / kafka_ex

Kafka client library for Elixir
MIT License
595 stars 162 forks source link

auto_commit not working in KafkaEx.stream/2 #145

Closed gjaldon closed 7 years ago

gjaldon commented 7 years ago

I'm using Kakfa 0.10.0 and have the following code:

for message <- KafkaEx.stream(@topic, partition) do
  process_message(pool, message)
end

The issue is that offsets are not being committed. I tried disabling the auto_commit option and then manually creating a worker and then manually committing the offset after the task but it still doesn't work. I based the offset_commit code on your test for the offset_commit function.

Also, process_message/2 above takes about half a minute to finish. In case, that matters.

What could be causing this problem?

bjhaid commented 7 years ago

@gjaldon sorry for the late response, have you tried KafkaEx.fetch do you have similar problems, I will attempt to reproduce sometimes today

gjaldon commented 7 years ago

@bjhaid no problem at all. I ended up getting the last offset from KafkaEx.fetch and passed the offset there to KafkaEx.stream as option so that it would start from that offset as a workaround.

spscream commented 7 years ago

try to use a named consumer group as a workaround, worked for us

dantswain commented 7 years ago

@gjaldon Can you provide a more complete reproduction case? I'd like to figure out what's going on here. Thanks!

dantswain commented 7 years ago

@bjhaid were you ever able to reproduce this?

bjhaid commented 7 years ago

I didn't try to :man_facepalming: