Open khairihafsham opened 2 years ago
Hi @amdeel , wondering if there is any update regarding this issue? This problem is blocking a big release of our service.
At the moment, we are exploring using External Events as an alternative way to "terminate" an orchestrator's execution. It is not ideal, but it does achieve the goal. Would appreciate if there is other solution that you could advice.
@davidmrdavid - Do you have any timeline/updates regarding when this change will make it to mainline and into a release? This bug is hammering us in production.
I see it currently sitting here: https://github.com/Azure/durabletask/tree/dajusto/patch-continue-as-new-and-terminate-race
@davidmrdavid - Still hitting this hundreds of time per day. Could you please give an update as I believe getting this into main is currently on you?
Description
When Continue-as-New and Termination happens about the same time, exception is thrown
System.InvalidOp erationException: Multiple ExecutionCompletedEvent found, potential corruption in state storage
It does depends on luck, but locally, I've managed to reproduce this issue quite often with the shared code below.
Expected behavior
Workflow is terminated
Actual behavior
Exception is thrown and Orchestrator retries exponentially to process both messages. The exception below is from my test using Durable Functions extension version 2.8.0
Relevant source code snippets
Note about the code, the timer value is set very low to increase the chances of reproducing the issue.
Known workarounds
Only tested locally, but triggering termination again via
func durable terminate --id uuid
manage to terminate the executionApp Details
Screenshots
If deployed to Azure
This issue is reproducible locally. But, here are some info that I can share.