swift-server / swift-kafka-client

Apache License 2.0
83 stars 24 forks source link

`KafkaProducer`: events- instead of acks sequence #96

Closed felixschlegel closed 1 year ago

felixschlegel commented 1 year ago

Motivation:

KafkaProducer: we want to expose a general KafkaProducerEvent type in the AsyncSequence that was formerly just for message acknowledgements. This enables us to add more events such as statistics in the future.

The reason behind having a single AsyncSequence for all event types instead of having separate AsyncSequences for each event type is that we need to ensure that all events that we subscribe to in librdkafka are actually consumed by the AsyncSequence in our KafkaProducer type. Otherwise we could run out of memory.

Now by giving the user the entire events AsyncSequence, they decide if they want to consume event or drop it.

Note: Logs will be consumed regardless of the KafkaProducerEvents AsyncSequence

Modifications:

felixschlegel commented 1 year ago

@swift-server-bot test this please