Acknowledgement for concrete message producing

[AlexeiNT]

Hi, want to use this client in my project, but have a problem. In my solution I want to ensure that concrete message is wrote to the Kafka broker. Current client's architecture provides only the possibility to subscribe on ProduceAcknowledged event, but information about acknowledgement is very poor for identification of what messages were produced. I see the ideal interface for produce something like Task ProduceAsync(...), do you plan to implement something like this in the ClusterClient?

[sdanzan]

This is not currenlty planned. If the ProduceAcknowledged event is not sufficient enough this is generally an indication one is usig Kafka a way it is not intended to be used. Could you tell me what exactly is your use case?

[AlexeiNT]

I have a distributed system with API for external clients. External clients can push a notification with specific data through this API. When my system recieves the notification it has to answer OK if the data is stored, (in kafka in my case), and ERROR if it is not. If a client receives ERROR it will resends the data later, if it is OK it will push the next notification. With asyncronous model of KafkaSharp produce aknowlagement, I dont know is notficiation stored or not, and lack of identification data in ProduceAknowledged event prevents the possibility to implement my own awaiter over the KafkaSharp client. I afraid that this asyncronous architecture of client can be the source of data loss, but data loss is unacceptable in my case.

[sdanzan]

If your're waiting acknowledegements for each message before processing the next client request Kafka is not a good match. You would be better using something like RabbitMQ. Kafka really shines at mass streaming messages but is really overkill (and not optimal) when you need to acknowledge messages one at a time.

That said you can try to do it by counting the number of pending messages and waiting for it to become 0 and reacting to errors. I don't know if it would fit your use case, it would probably cumbersome if you have multiple cients connecting to your API.

We could implement an acknowledged API (ProduceAsync) but I'm reluctant to do that because it would be easy to abuse and quite contradictory to the focus on performance (if you use that API on all messages, due to the batched nature of the underlying protocol it will means returning a lot of long lived future, which will increase the stress on the garbage collector.

On a final note, the official Confluent client (build on top the C client) should support what you need.