search

apache / pulsar-client-go

Apache Pulsar Go Client Library
https://pulsar.apache.org/
Apache License 2.0
659 stars 336 forks source link

Ability for a Consumer to bypass retries and DLQ a terminally failing message #1167

Open jcass8695 opened 9 months ago

jcass8695 commented 9 months ago

Is your feature request related to a problem? Please describe. If I consume a message and fail during the processing of that message in a terminal way, I know that I don't want to retry it, I want to send it directly to the DLQ. Currently, if I wanted a persistent record of the message, I would either have to

  1. ACK the message and log it, write it to a database etc.
  2. NACK the message and retry it up to the configured max retries, when it would then land in the DLQ.

Describe the solution you'd like I would like a consumer to be able to signal a message to be routed directly to the DLQ.

Describe alternatives you've considered See options 1 & 2 above.

Additional context Add any other context or screenshots about the feature request here.

RobertIndie commented 9 months ago

I think we could add an interceptor in the DLQPolicy so that we can let users choose whether each message need to be retried. Does it make sense to you?

Before implementing this, we need to follow the PIP process and get approval. Are you willing to lead this discussion? Would you like to start by sending the discussion to the mailing list? FYI, you can find the PIP process guide here: https://github.com/apache/pulsar/blob/master/pip/README.md

jcass8695 commented 9 months ago

I'm happy to lead the discussion on this. I agree that an interceptor in DLQPolicy looks to be a good route forward. I should have time to dedicate to it by the end of the week.

jcass8695 commented 3 months ago

PIP-367 PR created and sent to the mailing list for discussion.