Add ability to set offset in consumer group

[wtrocki]

To reply message/debug someone might need to get messages from begging or specific offset

[mblandgithub]

The docs say this: You can configure start offset for new consumers in settings (earliest, latest).

How do I go about view prior offsets in the tool? I can only view new messages that come after I start the consumer.

[wtrocki]

@mblandgithub I have implemented some POC using debezium (Kafka connect) for publishing and Node.js subscription resolvers for consuming. I haven't used GraphQL PubSub interface because it has some obvious problems.

Debezium will publish messages in order (partitioning is based on table/id)

POC does couple things

Producing

1. Debezium publishes events from datasource
2. Events are used as marker for change (not containing data itself for speed and low storage)

Consuming

1. New consumer group created everytime user connect (for each user). Client side need to always send unique ID)
2. Subscription requires sending offset from client. One can also persist offsets somewhere on backend but that will be less reliable to network glitches
3. Each message is send back with offset to client that keeps it in cache

This is complex so I'm trying to see if there is a way to make some simple sample app and blog post but not convinced yet that this is the most elegant solution (it is definitely opiniated one

[jlandersen]

I agree this would be nice. We can add an option to the consumers in the explorer to seek to a particular offset.

This might also be the opportunity to start documenting a bit on revamping the consumers. I've been wanting to redo the consumer part a bit for a while to make it a bit more flexible instead of just letting a single consumer group be registered. Similar to producers being created with a kafka file, i've been thinking of a similar approach to consumers. Something along the lines of this (needs some more thoughts but should show the gist of it):

CONSUMER test
topic: topic-a
partition: <number> (optional)
from: [earliest | latest | <offset>]
to: [latest | <offset>]

[angelozerr]

@jlandersen we discuss about this idea with kafka file with @fbricon but we wonder how to manage menu Start consumer from the treeview and kafka files (ex: menu from treeview item must introspect all kafka files in workspace to know which start consumer exists?)

That's why we think to manage those start consumer with a webwiew like I tried to explain in https://github.com/jlandersen/vscode-kafka/issues/88 liek Conduktor does:

Because there is not only offset confuguration, there is format configuration (to envoid encoding problem), etc

Perhaps we could manage start consumer with both solution (kafka file and webview), but I though next week to start to manage "start consumer) with webview. Are you OK with that?

[jlandersen]

@jlandersen we discuss about this idea with kafka file with @fbricon but we wonder how to manage menu Start consumer from the treeview and kafka files (ex: menu from treeview item must introspect all kafka files in workspace to know which start consumer exists?)

That's why we think to manage those start consumer with a webwiew like I tried to explain in #88 liek Conduktor does:

Because there is not only offset confuguration, there is format configuration (to envoid encoding problem), etc

Perhaps we could manage start consumer with both solution (kafka file and webview), but I though next week to start to manage "start consumer) with webview. Are you OK with that?

I wouldn't consider a consumer from a file the same as "Start consumer" from the menu, but rather they are two separate approaches to start / register a consumer.

• The quick right-click and Start consumer from menu i'd like to keep as an easy way to create a consumer for a particular topic
• Consumer from files is the more detailed approach to creating one, which can also be shared in a project repository, similar to producers. It is not related to a consumer started from the menu though. You could right click start consumer on topic A and create a consumer via file to consume topic A at the same time.

I'm definitely open to having a UI for starting a consumer as well, but functionally I'd expect it to work exactly the same as from whatever format we use in files (that is, any field you'd have in the UI would map to a property in the file format).

[angelozerr]

'm definitely open to having a UI for starting a consumer as well, but functionally I'd expect it to work exactly the same as from whatever format we use in files (that is, any field you'd have in the UI would map to a property in the file format).

Ok so I will give up webview UI stuff for the moment and I will start to work on kafka files with CONSUMER. @fbricon is it OK for you?

[angelozerr]

I'm working on start/stop CONSUMER with kafka file. See the demo https://github.com/jlandersen/vscode-kafka/pull/98#issuecomment-770436966

[angelozerr]

Fixed with https://github.com/jlandersen/vscode-kafka/pull/98 by using from into CONSUMER kafka file.

See documentation https://github.com/jlandersen/vscode-kafka/blob/master/docs/Consuming.md#kafka-file for more information.