DEndersby91
commented
9 months ago Amazing bug report @ranasch, I've been facing the exact same issue and had just made a striped down example and saw this when I came to post.
So it's not a complete waste of time here is the basic example I came up with : Functions V4, All latest relevant packages, .net8 Isolated, tested locally with Azurite.
using System;
using System.Threading.Tasks;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;
using Microsoft.DurableTask.Entities;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
public class EntityLockExample
{
[Function(nameof(EntityLockExample.EntityLockExampleTrigger))]
public async Task EntityLockExampleTrigger([HttpTrigger(AuthorizationLevel.Function, "get", Route = $"{nameof(EntityLockExample.EntityLockExampleTrigger)}/{{throwError}}")] HttpRequestData requestData, bool? throwError, [DurableClient] DurableTaskClient durableTaskClient)
{
await durableTaskClient.ScheduleNewOrchestrationInstanceAsync(new TaskName(nameof(this.EntityLockExampleOrchestration)), throwError.GetValueOrDefault(false));
}
[Function(nameof(EntityLockExample.EntityLockExampleOrchestration))]
public async Task EntityLockExampleOrchestration([OrchestrationTrigger] TaskOrchestrationContext orchestrationContext, bool throwError)
{
EntityInstanceId entityLock = new EntityInstanceId(nameof(EntityLockExample), "myLock");
await using (await orchestrationContext.Entities.LockEntitiesAsync(entityLock))
{
await orchestrationContext.CallActivityAsync(new TaskName(nameof(this.EntityLockExampleActivity)), throwError);
}
}
[Function(nameof(EntityLockExample.EntityLockExampleActivity))]
public void EntityLockExampleActivity([ActivityTrigger] bool throwError)
{
if (throwError)
{
throw new Exception("Activity Failed");
}
}
}With this code you can run it as many times as you want passing throwError = false But as soon as you pass one throwError = true the lock will never be released and all other subsequent requests will be held forever. The lock does know more and more tasks are in the queue though so that is getting updated.
I also tried to capture any exceptions inside the critical section and rethrow them outside the using block after the lock had been disposed but this sadly did not help.
Another "workaround" I have at the moment is to never allow an orchestration that uses critical sections to enter a failed state. To do this I capture any exception in the activities and store that in a custom state. This is less than ideal though as using the DFMonitor it looks like everything is running fine when really there may be many issues.
I have read some other tickets related to this issue and people where asking about the Netherite storage engine which I hoped would solve this issue but sadly I can't get this to work with .net8 Isolated. Might be uncouth but here is a link to that issues in the Netherite repo.
Thanks @sebastianburckhardt for already assigning this to yourself, if you have any other ideas or work around that would be ace.
sebastianburckhardt
ranasch
roend83
Description
Running .net8 isolated durable function which has a critical section guarded by a durable entity. When the orchestration fails, the entity keeps locked and no further orchestrations can proceed.
Expected behavior
The the critical section (using) is left via exception, the build in dispose should release the lock of the orchestration on the entity, otherwise it stays locked forever and no other orchestration can obtain a lock.
Actual behavior
When the critical section is left via exception, the lock is not released on the entity, because the release is not persisted on the Azure Table.
Relevant source code snippets
The created entity will stay be locked by the critical section, unless you also implement a custom Dispose, which by calling the Entity a final time ensures the unlock state is persisted in the table storage:
Unfortunately this approach has a catch with throwing ambiguous workflow exception. So its not the solution.
Known workarounds
Tried the custom dispose, however that throws this exception:
App Details
Project:
settings:
Screenshots
Lock before disposing:
Entity unlocked after custom callout in dispose:
If deployed to Azure
This is a simplified excerpt of our production function. For production details reach out to me directly to get a repro.