search

eventuate-foundation / eventuate-messaging-kafka

Other
15 stars 11 forks source link

Is there an option to slow down the pulling of messages from kafka topic when the consumer queue is full. #51

Open ddaeyongee opened 2 years ago

ddaeyongee commented 2 years ago

hi. My name is Taeyong Kang. I am working in the software architecture organization of LG CNS. I have a question while doing this MSA project, so I'm posting an issue.

I am looking at the potential issue of back-pressure i.e. fast producer and slow consumer. In this case, producer will be the component that will read messages from kafka topic. Is there an option to slow down the pulling of messages from kafka topic when the consumer queue is full. If yes, how is it effective to apply?

To implement this, I am using TPL data flow BufferBlock class. BufferBlock has a bounded capacity. If the component is not able to post messages to the buffer block, I want to slow down the kafka component that is pulling messages from topic. Please advice!

cer commented 2 years ago

See

https://github.com/eventuate-foundation/eventuate-messaging-kafka/blob/2e95af2cb90a59802398ea8c4c0f0638e348cb3d/eventuate-messaging-kafka-spring-basic-consumer/src/main/java/io/eventuate/messaging/kafka/spring/basic/consumer/EventuateKafkaConsumerSpringConfigurationProperties.java#L13

and

https://github.com/eventuate-foundation/eventuate-messaging-kafka/blob/2e95af2cb90a59802398ea8c4c0f0638e348cb3d/eventuate-messaging-kafka-integration-test/src/main/resources/application.properties#L12-L13

The goal of the BackPressureConfig mechanism to 'slow down' the consumer so that it doesn't read (into memory) a large number of messages from a topic.