Receiving messages has side effects with offsets to commit

[danielmarbach]

It might be that I don't yet fully understand the usage of the kafka clients. When I was looking at

https://github.com/pablocastilla/NServiceBus.Kafka/blob/master/src/NServiceBus.Kafka/Connection/ConsumerHolder.cs#L206

I had some spider senses tingling. For me the code read like

Whenever a message is completed we loop through OffsetsReceived https://github.com/pablocastilla/NServiceBus.Kafka/blob/master/src/NServiceBus.Kafka/Connection/ConsumerHolder.cs#L194 and commit all those that can be completed, essentially batching offsets to complete together but only triggered due to processing of messages.

Wouldn't it be more safe to do the following:

• Only complete things that can be truly associated with the message received

or apply smart background batching similar to what we do in Azure Service Bus.

https://github.com/Particular/NServiceBus.AzureServiceBus/pull/384

[pablocastilla]

The overall idea is to commit only messages that can be committed but without any hole in the sequence.

If we have launched offsets: 15,16,17,18 and we receive and ok for 17 we have to wait until 15 and 16 are processed. That's what is done there. This has to be done per topic/partition

I suppose this processing can be decoupled and done through a timer each second (configurable) or something like that.

How do you see this approach?