reisenberger
commented
5 years ago Thanks @george-polevoy for the question around Polly and durable functions (sorry for delayed reply).
TL;DR You can use Polly in durable Activity functions as-is, but not in Orchestration functions.
My understanding is that the need for determinism in durable functions and the consequent time-provider recommendations apply only to durable Orchestrator functions as these are the type of durable function that is replayed. But the kinds of operation that Polly normally guards - any external I/O, any async HTTP calls etc - are specifically barred from use within orchestration functions.
On the other hand, durable Activity functions have no such constraints, and this is where I/O, HTTP calls etc should be placed. As activity functions are not replayed and don't have the same constraints, Polly should be perfectly safe to use (without modification) within Activity functions.
The Activity functions themselves which the orchestrators coordinate, could of course succeed or fail, so one could ask: is it possible to apply Polly-like controls to the way orchestrators execute activity functions? In most cases, durable functions provides equivalent functionality.
Retry: Durable functions provides its own in-built retry functionality including backoffs.
Timeout: Durable functions provides its own timeout pattern recommendation (example 1; example 2), analogous to Polly's pessimistic timeout.
Cache: Orchestrator functions already cache the results of activity functions.
Bulkhead isolation: is essentially a parallelism throttle. Durable functions provides its own concurrency throttles and scaling controls.
Circuit Breaker: is essentially about accumulated errors over time or across parallel processes. But orchestrator functions are isolated, single-thread processes - the only way they could accumulate success/failure statistics cross-process would be in some external store, but orchestrators are not allowed to communicate externally anyway (except via Activity functions) ... so the problem becomes circular. Dealing with replays through a circuit-breaker in an orchestration function would also present a challenge - when the orchestration was in replay state, the circuit-breaker would need to record neither a success or failure - because both influence statistics and the information would be in any case outdated. It's not impossible, but TL;DR it would be much simpler to simply push the circuit-breaker down into the Activity function. This is probably also only useful when distributed circuit-breaker becomes available.
Fallback and PolicyWrap would work in orchestrators - but on their own (without the other policies) they are not that relevant.
Hope this helps. Interested if anyone can see angles on Polly-type behaviour in durable functions that this brief assessment misses.
TL;DR Use Polly in the Activity functions rather than the Orchestration functions.
This a feature suggestion
In Azure durable functions, I need a policy which is completely deterministic, because it could run the same code while replaying past events, and policy must yield reproducible flow of execution. This means we can't use system clock and entropy (and any other kind of kernel resources for that matter).
For a policy to be deterministic and reusable inside the same process we need to use an instance of clock service, not a static function such as SystemClock.SleepAsync, which uses system clock internally.