Feature/concurrency exception handling

[rhyslbw]

Implements at least one aspect of dealing with concurrency issues, where we determine that the commit trying to be made is now out-dated.

The router now tries to save the commit using the repository, and if a concurrency error is thrown out of the commit store, the message is passed back into the handler for re-processing, where it could either be accepted by the aggregate, or throw a domain exception if the state has changed that renders the message invalid. That is an application concern then.

There are still concurrency opportunities closer to the database, but this is at least a good start, and set’s the groundwork for other exceptions to be managed at the place where we still have the message and can deal with it appropriately. Done!

[rhyslbw]

Forgot to add, this includes the index specified in #82

[rhyslbw]

@qejk Ah so we could actually also retry the errors that happen closer to the database too, like you documented in #75, as this is also a rejection of the same nature. I'll throw the same error at this point, as it's the same situation, just different layers picking it up.

[rhyslbw]

Ok @DominikGuzei, ready for review

[rhyslbw]

I've just run two local load tests with this branch and a scenario of 5 and 25 concurrent operations. Both we're 100% successful:

Details

Macbook Pro 16GB with local RabbitMQ 1905 items items in queue Both tests had 0 permanent failures, verified by a projection which tests the commit publishing too. No apparent looping, just a slower rate of acceptace depending on the number of concurrent operations (as expected) 0 failed commit processing operations Query used (bonus data point)

db.getCollection('space_eventSourcing_commits').find({
  receivers: { $elemMatch: { failedAt: { $exists: true }}}
}).count() // Result = 0 

Failures that were eventually handled (logged as warnings)

• 5x: 610
• 25x: 15 962

Summary

• This was a sustained constant load, and and IMO performed really well. Even at 25x the commits were being accepted at a rate of 7/sec
• It's highly unlikely a single instance would be dealing with this many commits, so horizontal scaling is the natural approach to dealing with load

cc @sanjo, @samhatoum

[DominikGuzei]

Kickass work @rhyslbw! Thanks for pushing this through :+1: I cannot test it atm but i trust that you did thoroughly!