skroutz / rafka

Kafka proxy with a simple API, speaking the Redis protocol
https://engineering.skroutz.gr/blog/kafka-rails-integration/
GNU General Public License v3.0
8 stars 0 forks source link

consumer: Drop invalid offsets from OffsetsCommitted output #79

Closed dblia closed 5 years ago

dblia commented 5 years ago

When a Rafka consumer receives an OffsetsCommitted event it will log an array with information about the commited offsets. This array contains information about all partition/offset pairs in which the Rafka consumer has been subscribed to, regardless of the amount of data that have been actually commited by librdkafka.

For example, consider a topic with 12 partitions; the OffsetsCommitted event of a single partition/offset pair (part: 9, offset: 1), looks like the following snippet:

[cgroup|cid|topic] 2019/08/02 13:43:36 OffsetsCommitted (<nil>, [topic[0]@unset topic[1]@unset topic[2]@unset topic[3]@unset topic[4]@unset topic[5]@unset topic[6]@unset topic[7]@unset topic[8]@unset topic[9]@1 topic[10]@unset topic[11]@unset])

This extraneous information makes debugging quite difficult; we actually interested in the updated partitions and not about the unaffected ones.

This commit introduces a new utility named dropInvalidOffsets which will be used to trim the "unset" offsets (aka those of OffsetInvalid type) from the logging output.

The logging ouput of the OffsetsCommitted event for the aforementioned example will be simplified as presented below:

[cgroup|cid|topic] 2019/08/02 13:42:20 OffsetsCommitted (<nil>, [topic[9]@1])