spring-attic / spring-cloud-gcp

Integration for Google Cloud Platform APIs with Spring
Apache License 2.0
704 stars 694 forks source link

Unable to batch outgoing messages with PubSub publisher properties #2621

Closed leee0n closed 3 years ago

leee0n commented 3 years ago

Been recently trying to add the batching features available for GCP PubSub and so far I can't see any change in the behavior. As soon as the message gets into the service it goes out to the PubSub immediately without waiting for the conditions to be satisfied (5 elements processed or time spent higher than delay threshold after first processed message).

Just added the following to my application.properties file.

spring.cloud.gcp.pubsub.publisher.batching.enabled=true
spring.cloud.gcp.pubsub.publisher.batching.delay-threshold-seconds=100000
spring.cloud.gcp.pubsub.publisher.batching.element-count-threshold=5

My existing properties are as follows.

spring.cloud.stream.function.bindings.myMethod-in-0=input
spring.cloud.stream.function.bindings.myMethod-out-0=output

spring.cloud.stream.bindings.output.destination={$OUTPUT}
spring.cloud.stream.bindings.output.binder=pubsub
spring.cloud.stream.bindings.input.destination={$INPUT}
spring.cloud.stream.bindings.input.binder=pubsub
spring.cloud.stream.bindings.input.group={$SERVICE}

Is it possible that I'm missing something? Do I require any extra properties?

Also, do the properties help to change the output type to a List<?> instead of a single object instead? Or that would be a task for the developer to code it within the service itself?

elefeint commented 3 years ago

Try setting spring.cloud.gcp.pubsub.publisher.batching.request-byte-threshold to a reasonable number -- by default, the client library sets it at only 1 byte, which effectively turns off batching.

I'll update the documentation to reflect this correctly.

leee0n commented 3 years ago

Works flawlessly @elefeint , thanks a lot!