search

absmach / magistrala

Industrial IoT Messaging and Device Management Platform
https://www.abstractmachines.fr/magistrala.html
Apache License 2.0
2.48k stars 674 forks source link

Add support for dead letter in event log #652

Open anovakovic01 opened 5 years ago

anovakovic01 commented 5 years ago

FEATURE REQUEST

  1. Is there an open issue addressing this request? If it does, please add a "+1" reaction to the existing issue, otherwise proceed to step 2. No.

  2. Describe the feature you are requesting, as well as the possible use case(s) for it. There is a possibility that some events from Redis Streams cannot be processed. In this case, currently, our consumer will be stuck on this event, which is not good. We should send such events to some other stream where we'll store problematic events. This way it will be easier to track such events and also, our consumers won't get stuck on such messages.

  3. Indicate the importance of this feature to you (must-have, should-have, nice-to-have). This is a must-have feature.

nmarcetic commented 5 years ago

@anovakovic01 Can you make example of such an event, when this can occur? A consumer should handle the processing and catch errors or I am missing a point? Maybe simple ERROR processing failure logging will be enough? I don't get it how We should send such events to some other stream where we'll store problematic events can help to not stuck the consumer?

anovakovic01 commented 5 years ago

@nmarcetic Here is nice explanation from Redis Streams documentation.

The counter that you observe in the XPENDING output is the number of deliveries of each message. Such counter is incremented in two ways: when a message is successfully claimed via XCLAIM or when an XREADGROUP call is used in order to access the history of pending messages. When there are failures, it is normal that messages are delivered multiple times, but eventually they usually get processed. However there is sometimes a problem to process a given specific message, because it is corrupted or crafted in a way that triggers a bug in the processing code. In such a case what happens is that consumers will continuously fail to process this particular message. Because we have the counter of the delivery attempts, we can use that counter to detect messages that for some reason are not processable at all. So once the deliveries counter reaches a given large number that you chose, it is probably wiser to put such messages in another stream and send a notification to the system administrator. This is basically the way that Redis streams implement the concept of the dead letter.