search

wvanbergen / kafka

Load-balancing, resuming Kafka consumer for go, backed by Zookeeper.
MIT License
373 stars 141 forks source link

Handle sarama.ErrOffsetOfRange error somewhat gracefully. #57

Closed shivnagarajan closed 9 years ago

shivnagarajan commented 9 years ago

When the offset retrieved from initializePartition in offsetManager, is very far off from the currently available offset, (i.e. < offsetOldest or > offsetNewest) the consumePartition call will return a sarama.ErrOffsetOutOfRange error.

This change just adjusts the offset to either offsetOldest or offsetNewest, based on what was specified in the sarama.Config and retries the call to consumePartition

@wvanbergen

Will write a specific test to test out this behaviour also, just wanted to get early opinions :smile:

wvanbergen commented 9 years ago

This looks good to me :+1:

As for testing: unit testing is quite hard right now, but I think you should be able to test this with a functional test by writing a negative offset to Zookeeper (don't use -1 or -2, lower is fine).