search

kgrzybek / sample-dotnet-core-cqrs-api

Sample .NET Core REST API CQRS implementation with raw SQL and DDD using Clean Architecture.
https://www.kamilgrzybek.com/design/simple-cqrs-implementation-with-raw-sql-and-ddd/
MIT License
2.86k stars 641 forks source link

If there have multiple instances of ProcessOutBoxJob, The events may be processed in many times. #15

Closed zh6335901 closed 4 years ago

zh6335901 commented 4 years ago

ProcessOutboxJob retrieved the unprocessed events then process and set processed flag, It work fine under single instance situation. But if there have multiple instances of ProcessOutboxJob(In different processes), The events may be concurrently retrieved many times, Then process many times. Do you have any idea resolve it?

kgrzybek commented 4 years ago

Hi @zh6335901,

If you have many Outbox processors you can't process events in parallel because the order of events matters. It will be a problem for your subscriber if he would receive OrderCanceled event and then OrderCrated event.

Even you will solve the problem with processing one event many times using some locking mechanism on the table or more advanced solution like using Rabbit MQ which has "round-robin" by default, the order of published events would be incorrect.

So you should have only one processing instance working at the same time. If you want to have multiple instances, other instances should be inactive. You can achieve this using Single Consumer Rabbit MQ feature: https://www.rabbitmq.com/consumers.html#single-active-consumer As documentation says:

Consuming with only one consumer is useful when messages must be consumed and processed in the same order they arrive in the queue.

zh6335901 commented 4 years ago

Hi @kgrzybek You are right. The order of events is important. But if the Outbox processors only have one instance, I am worried about it would be lost the high available. Maybe i need find a solution that multiple instances but only one working at the same time.

zh6335901 commented 4 years ago

Thank you for your answer. And your aritcles is help me out