search

spring-attic / spring-cloud-gcp

Integration for Google Cloud Platform APIs with Spring
Apache License 2.0
704 stars 694 forks source link

Spring Cloud Stream Pubsub's Equivalence of prefetch count in RabbitMq #2565

Closed thayhoang closed 3 years ago

thayhoang commented 3 years ago

Does Pubsub have something similar to the prefetch count in rabbitMq?

"The prefetch value is used to specify how many messages are being sent to the consumer at the same time. It is used to get as much out of your consumers as possible."

I'm looking at the spring cloud pubsub properties and don't find any settings similar.

elefeint commented 3 years ago

For similar behavior, you could experiment with spring.cloud.gcp.pubsub.subscriber.flow-control.max-outstanding-element-count and spring.cloud.gcp.pubsub.subscriber.flow-control.max-outstanding-request-bytes properties. I would stick with defaults, though, unless you observe a problem that needs solving, though -- Pub/Sub streaming is fairly well optimized for high volumes by default.

thayhoang commented 3 years ago

Thanks you, @elefeint The document about max-outstanding-element-count seem not clear to me: Maximum number of outstanding elements to keep in memory before enforcing flow control.. What's outstanding element and what's flow control?

thayhoang commented 3 years ago

@elefeint Correct me if I'm wrong. Suppose I set the parallel-pull-count = 1, my app will open 1 connection to pubsub to get messages. And if I set the max-outstanding-element-count = 10, my app will receive max of 10 messages before pausing the message stream

elefeint commented 3 years ago

@thayhoang I checked with the Cloud Pub/Sub team, and your understanding is correct. Flow control is a way for the client application to provide hints to the server about how many messages to send. Outstanding elements are the ones that have been sent by the server, but have so far been neither acked nor nacked by your application.