search

rebus-org / Rebus

:bus: Simple and lean service bus implementation for .NET
https://mookid.dk/category/rebus
Other
2.26k stars 353 forks source link

Deduplication of Saga launches and other handlers in the "idempotent Saga". #1136

Open glazkovalex opened 5 months ago

glazkovalex commented 5 months ago

Hellow, @mookid8000! When a single handler is idempotent, everyone expects that it can be launched with the same parameters many times and at the same time be sure that repeated launches will not unnecessarily affect half of the services in information systems and will not spoil business data. The current implementation of the "idempotent Saga" in Rebus, regardless of transport, is idempotent only in relation to the internal status state of its states machine, but NOT the EVENTS generated by it. That is, the current implementation of the "idempotent Saga" in Rebus is NOT idempotent in relation to the business process it implements due to the following features of its implementation:

  1. An "idempotent Saga" is not idempotent with respect to the first event triggering it. This leads to the cascading launch of duplicate Sagas and repeated repetitions of all its steps, and also bothers technical support with multiple messages "Could not find existing saga data for message..." and warnings "Unhandled exception 1 while handling message with ID ...". This is a very big drawback, especially considering that some programmers who do not know that there should never be business-logics and business data in the Saga can add business logic to the Saga and temporarily store business data in it, without providing for the idempotency of this business logic. This flaw in the Rebus Saga is very easy to fix. To prevent this drawback, you can provide an optional "IdempotentSagaBehavior.StartsDeduplication" parameter, if available, DO NOT delete the IDs of previously launched Sagas from the "SagasIndex" table and ignore repeated attempts to launch duplicates.
  2. The "Idempotent Saga" is not idempotent with respect to all its other message handlers. In the case of delivering a message with the same "rbs2-msg-id" and "rbs2-corr-id", Saga will re-process the message a second time, calling all services with business logic a second time, and send all messages generated by it again. Considering that the Saga already stores all the information about the "rbs2-msg-id" and "rbs2-corr-id", it would be great if idempotents not only knew the status of the Saga, but also the result of its execution. To implement this, it is enough to add the optional parameter "IdempotentSagaBehavior.StartsAllSagaHandlersDeduplication" and provide in "SagasIndex" in addition to "rbs2-corr-id" to save a json array of "rbs2-msg-ids" already processed by it and skip repetitions without reprocessing.

It is clear that the Saga cannot ensure the delivery of its messages only once and repeated deliveries of its messages are possible in any case. Therefore, all her clients should have idempotent handlers for her messages. But generating a batch of repeated messages with a lot of alarming messages and warnings is not at all what users expect from an "IDEMPOTENT" Saga.

As a general alternative solution to the problem of message deduplication, it would be possible not to change the "idempotent Saga", but to implement a separate package for custom deduplication of the necessary messages by "rbs2-msg-id", "rbs2-corr-id" and other message headers. I am sure that such a universal extension would be in great demand, since in almost every system it is necessary to implement deduplication of some messages.

Such a refinement of the "idempotent Saga" would greatly simplify the use of the "Idempotent Saga" of Rebus and would meet the expectations of its users.

glazkovalex commented 4 months ago

The best way to implement message deduplication I have seen in the "Azure Service Bus" If you make a dynamically configurable "window" for each topic separately, it will be optimal in terms of ease of use, maximizing performance and minimizing duplicates. Such a "window", which optionally would allow you to automatically send "rotten" messages older than the specified age inside the "window" of duplicates, immediately to the error queue.

The user interface is, of course, superfluous. Common global default settings, redefined for individual messages via the configuration (IOptions) of the Rebus message deduplication extension, would be sufficient.

Well, in the meantime, there is no such beauty in the Rebus, you will have to deduplicate the entire message, as always, using clumsy grandfather's methods 🙂