Kafka starter - insufficient doc.

[mocenas]

Kafka is a powerful and complex messaging system. But our current starter and it's docs shows only a simple scenario for basic consumer - producer model.

The docs should contain also (maybe it will take an addition to API too):

• Publish-subscribe model - how to use it.
• Working with Kafka partitions. Current API have methods using the partitions, but nowhere is written how to use it.
• Replaying old messages. Or generally random-access to messages in topic. This is one the most highlighted Kafka features.

[gytis]

@mocenas I've expanded the README https://github.com/snowdrop/vertx-spring-boot/commit/5d84233731c1c010d807585c562f8b97ebe272c3. Please take a look and let me know if it now answers your questions.

[mocenas]

@gytis Thanks, it's much better now. But still some things are missing. The more important ones:

1. Producer-consumer (or queing) vs publish-subscribe model.
   • Producer-consumer (or queing) means producer sends message, and exactly one consumer at the topic will receive it.
   • Publisher-subscribe means publisher (or producer) sends message and all consumers (or subscribers) on the topic will receive it.
   • I tried to figure out if it can be configured on the client or needs to be done at the cluster but no luck so far. But i guess it can be done at the client since Kafka supports auto creating topics. This should be somehow explained in the doc.
2. consumer.flux.subscribe(record -> {}) What data type is the record? How to get the message from it?

Less important ones:

• Can I somehow find what partition am I on? or which partitions do exist?
• In consumer.seek(partition, offset) is the offset absolute or relative? Can I somehow get my current offset?

[gytis]

1. This a generic Kafka theory not dependant on a client API that is being used. You setup your application to be producer-consumer or publisher-subscriber using the consumer groups. With this this particular client library you can do that in your application configuration (this will set the same group id for all your consumers) or when creating a consumer.

2. A consumer interface has genetic types (https://github.com/snowdrop/vertx-spring-boot/blob/master/vertx-spring-boot-starter-kafka/src/main/java/dev/snowdrop/vertx/kafka/KafkaConsumer.java). You have to specify what type of records will be consumed when creating a consumer.

3. There is assignments method for that.

4. I'm not sure what is a relative offset. Every consumer tracks an offset of where it is in a partition. So if you seek to an offset 0 you'll end up at the start of the partition.

[mocenas]

1. Yes this is generic Kafka theory. But I take this Kafka starter as a blackbox (as probably customers will). You say I can setup producer-consumer or publisher-subscriber via application configuration - I ask how? what properties to set? Is Kafka starter a wrapper around some generic Kafka client, whose properties can I use? I understand we can't cover everything in the doc, but this is really basic use case, which I really should be able to handle, just by reading the doc.

2. Ok, can you at least link KafkaConsumerRecord in the doc and explain it a bit? Having current consumer.flux().subscribe(this::doSomething); is really not helpful enough.

3. OK

4. By relative I meant, relative to my current offset - e.g. I'm on offset 10, .seek(-3), I'm on offset 7. The way it is in the doc is pretty much OK, I was rather curious.

[gytis]

Added a background section with links to the Kafka and upstream client docs https://github.com/snowdrop/vertx-spring-boot/commit/4bc3ff12e28fd5b47ebddf00c70191c962e1a60f