search

phaistos-networks / TANK

A very high performance distributed log service
Apache License 2.0
938 stars 70 forks source link

Consumer Offsets, support via internal topic/partition #40

Open markpapadakis opened 8 years ago

markpapadakis commented 8 years ago

According to @miguno ( I need to go through the Kafka codebase to figure out what they do exactly ), they are maintaining an internal/special-purpose topic __consumer_offsets, where the policy is configured as cleanup, because they only really care for the last value, and, presumably, on startup, the partition is replayed in order to restore the state.

That's a clever and pragmatic way to implement it.

See also: http://www.slideshare.net/jjkoshy/offset-management-in-kafka

markpapadakis commented 6 years ago

See: https://www.ctheu.com/2017/08/07/looking-at-kafka-s-consumers-offsets/

A new topic_partition::foreach_msg() has been implemented and can be used to effortlessly iterate across all messages, both in the RO segments and the current RW segment. This can be used to materialize state by iterating all messages in, say, __consumer_offsets partitions.