Implement circuit-breaker functionality

[fgheysels]

Is your feature request related to a problem? Please describe. When you have a message-handler that has a dependency on an external resource, that resource might become unavailable and in that case we want to temporarily stop processing messages.
For this, we can use the MessagePumpLifeCycle which is already available in Arcus.Messaging.

However, I'd like to avoid that once the 'pause-time' has elapsed, we start processing a whole bunch of messages again. It would be better to try processing messages at a slower rate and only once the downstream-dependency is fully functional again, we can resume with processing larger batches.

Describe the solution you'd like The circuit breaker pattern is described here. The 'half-open' state is particularly interesting.

Can we implement this by building further on the MessagePumpLifeCycle ?
Just some random thoughts:

• We could specify -via a property in MessagePumpLifeCycle- the amount of messages that should be retrieved once the 'pause duration' has elapsed. When this property is set to 1, then once the pause-duration has elapsed, only one message will be retrieved from the queue / topic. This allows the process to try and see if processing can continue by processing a single message.
• Next to the property described above, maybe an additional interval can be specified which defines how long the pump should wait between retrieving messages when we're in the 'half-open' state. (When trying, we want to process maybe a single message at a time, and also wait a few seconds before processing the second message, .... )
• The MessagePumpLifeCycle could keep track of counters that keep track of the amount of messages that have been processed (succesfully and failed). (I think it should be the responsability of the 'processor' to increment these properties; maybe via a method call on the MessagePumpLifeCycle which does that ?).
• Maybe MessagePumpLifeCycle must be renamed to CircuitBreaker (or something similar).

[stijnmoreels]

This looks like a cool feature, but some further investigation will be required indeed to look for the possibilities. We could use the current 'life-cycle' functionality, but then we should 'start fresh' with a new interface and deprecate this old one for backwards compatibility.

[stijnmoreels]

@fgheysels Also a point worth noting: this is a feature of the pump (receiver) not of the processor (router), so it will not be available for Azure Functions applications as in those cases the Trigger acts as the receiver.

[fgheysels]

@fgheysels Also a point worth noting: this is a feature of the pump (receiver) not of the processor (router), so it will not be available for Azure Functions applications as in those cases the Trigger acts as the receiver.

That's something that must be documented then. We cannot control the behavior of the ServiceBus Trigger in Functions

[fgheysels]

This looks like a cool feature, but some further investigation will be required indeed to look for the possibilities. We could use the current 'life-cycle' functionality, but then we should 'start fresh' with a new interface and deprecate this old one for backwards compatibility.

Or we increase the major version-number to make it clear that this is a breaking change ...