search

apache / pulsar-client-go

Apache Pulsar Go Client Library
https://pulsar.apache.org/
Apache License 2.0
660 stars 335 forks source link

Return an error when closing the client/producer/consumer fails #1063

Open RobertIndie opened 1 year ago

RobertIndie commented 1 year ago

Is your feature request related to a problem? Please describe. If the producer/consumer/client fails to close, the go client only logs the error instead of returning to the user. https://github.com/apache/pulsar-client-go/blob/e45122c2defc5efd4efc493d0acef278a7ccfc01/pulsar/producer_partition.go#L1295-L1297

Describe the solution you'd like Add the return type to the Close() method:

Close() error

If there are any errors, return back to the user's code.

gunli commented 1 year ago

Hi @RobertIndie , I thought of a question last night, When we close the client and fail the messages that are pending, will that be a question when message is chunked?

What I mean is when a BIG message is spilt into many chunks, some chunks have been sent and some are still pending. An extreme case, if the last chunk is sent and some other ones are pending, the callback of the message won't get called, 'cause we only trigger the callback for the last chunk, it seems we will lost the last chance to save this failed message.

And at the server side, will that be a question if only some chunks are received?

RobertIndie commented 1 year ago

Hi @gunli

if the last chunk is sent and some other ones are pending

I don't think that would happen. The message order in a single topic producer is guaranteed. If the last chunk is sent successfully, then other ones must be sent.

For instance, if we have a large message with 3 chunks(M0 to M2). We have sent M0. But M1 and M2 are pending. If we fail them, then the whole large message is failed to send. This behavior is safe because the consumer will skip this corrupt message. But it will never be the case that, for example, M2 sends successfully but M1, and M0 fail.

And at the server side, will that be a question if only some chunks are received?

The broker will only treat the chunk as a normal message. And the consumer will skip these chunks if it couldn't get the last chunk of the message.

For more about how chunking works, you could check this blog: https://streamnative.io/blog/deep-dive-into-message-chunking-in-pulsar#best-practices-for-message-chunking

gunli commented 1 year ago

Good. So we can return the failed messages to the users and let them decide what to do.