search

spring-attic / spring-cloud-gcp

Integration for Google Cloud Platform APIs with Spring
Apache License 2.0
705 stars 693 forks source link

Spring Cloud Stream Gcp does not retry #2406

Open limyongquan1992 opened 4 years ago

limyongquan1992 commented 4 years ago

Hi, I am following spring-cloud-gcp-pubsub-stream-binder-functional-sample. But I find that the retrying as per the Spring Cloud Stream documentation does not work.

What I have Tried

  1. Manually declare a StreamRetryTemplate bean - not working.
  2. Try out different configs for my consumer (such as changing the spring.cloud.stream.bindings.input.consumer.xxx, spring.cloud.gcp.pubsub.subscriber.retry.xxx, etc.) - not working.
  3. Test with Kafka Stream Binder - the retry is working out of the box.

Is there any extra configuration needed to switch on this feature on GCP PubSub?

elefeint commented 4 years ago

Sorry for the delay! There are two possible retry scenarios: 1) Retry if message retrieval from Pub/Sub failed 2) Retry if the application message processing logic failed

Looking at Spring Cloud Stream code, (1) a retry template is created automatically for pollable consumers. Our Cloud Pub/Sub adapter does give ability to create a pollable source, but the streaming consumer is more commonly used.

It seems that your requirement is to do (2) -- retry if downstream processing failed. Kafka stream binder sets up its own retry template infrastructure to do that, but Spring Cloud GCP does not provide the same instrumentation.

We are happy to accept retry template for downstream processing logic as a feature request for Spring Cloud GCP. In the meantime, you can accomplish the same thing by creating a custom RetryTemplate and using it in your message processing logic.

@olegz Is my understanding correct that Spring Cloud Stream itself does not retry message processing functionality? The documentation seems to imply that, but I only see Kafka implementing retry on user logic.

olegz commented 4 years ago

@elefeint we do use RetryTemplate for that. There are details here NOTE: This does not apply to reactive consumers (e.g.Function<Flux, Flux>) Let me know if I can help

elefeint commented 4 years ago

@olegz Could you point me to code in Spring Cloud Stream? Sorry, I find code easier to read than documentation and, ah, more trustworthy :) I see retry template being bound to a pollable consumer here to retry message retrieval. But the messaging infrastructure comes from Spring Integration, so retry template has to be bound to the underlying object somewhere for the non-pollable, and I don't see where.

Spring Kafka's RetryingMessageListenerAdapter implements its own retry of downstream consumer logic here. I can see that it's useful to do something like this in Spring Cloud GCP.

artemptushkin commented 3 years ago

@elefeint any update on this? Was feature request created? I wondering about retrying application message processing too

ttomsu commented 3 years ago

@artemptushkin - I don't think any progress has been made here. Would you be willing to contribute a patch to fulfill the feature?

artemptushkin commented 3 years ago

@ttomsu frankly, I would like to. I just started using spring-cloud-gcp stack in my company and it seems convenient to work on it, as I see and other small places for improvements. I'll create a task with the intention

ttomsu commented 3 years ago

@artemptushkin I suggest working with GoogleCloudPlatform/spring-cloud-gcp and we'll back port the fix if necessary.

dzou commented 3 years ago

Hey @artemptushkin, I duplicated this issue in our new repo so we don't forget about this: https://github.com/GoogleCloudPlatform/spring-cloud-gcp/issues/337

We can continue conversations there