IDurableOrchestrationContext.CallSubOrchestratorAsync is missing events

[MhAllan]

Description

In the code below I have this object class Payment { string PaymentMethod }. It is coming from CosmosDB to a singleton orchestrator which calls sub orchestrator. when I change the object in Cosmos DB, Cosmos DB triggers and also the singleton orchestrator. The sub orchestrator triggers as well for the first change. Now, if I change the object again while the sub orchestrator is busy then the sub orchestrator continues with the previous value and drops the new value event.

Expected behavior

The sub orchestrator to resume with the current execution then pick up the next message

Actual behavior

The sub orchestrator resumes with the current execution but drops the next message

Relevant source code snippets

[FunctionName(nameof(SingletonOrchestrator))]
    public async Task SingletonOrchestrator([OrchestrationTrigger] IDurableOrchestrationContext context)
    {
        try
        {
            var payment = await context.WaitForExternalEvent<Payment>(PaymentEvent);
            var orchId = payment.Id;
            await context.CallSubOrchestratorAsync(nameof(NotificationSender), orchId, payment);
        }
        catch(Exception ex)
        {
            _logger.LogError(ex.Message, ex);
        }
        finally
        {
            context.ContinueAsNew(null);
        }
    }

    [FunctionName(nameof(NotificationSender))]
    public async Task NotificationSender([OrchestrationTrigger] IDurableOrchestrationContext context)
    {
        var payment = context.GetInput<Payment>();
        if(payment.PaymentMethod == "visa")
        {
           //introduce delay so the two messages overlap
            await context.CreateTimer(context.CurrentUtcDateTime.AddSeconds(20), default);
        }
        _logger.LogCritical(JsonConvert.SerializeObject(payment));
    }

Known workarounds

No workarounds

App Details

<PackageReference Include="Microsoft.Azure.WebJobs.Extensions.DurableTask" Version="2.6.1" />
<PackageReference Include="Microsoft.NET.Sdk.Functions" Version="4.1.0" />

Environment

• Function .Net 6
• VS 2022
• Cosmos DB Emulator (triggers the singleton orch)
• Storage Emulator

[cgillum]

This behavior is by design - the second "start" event for the sub-orchestration is considered duplicate and discarded.

Is there any reason you can't use a unique instance ID for the sub-orchestration?

[MhAllan]

@cgillum it is not duplicate, here is timeline

• Set cosmos db value to X
• Sub-Orch starts handling X
• Set cosmos db value to Y
• Sub-Orch stilling handling X
• Sub-Orch finishes with X
• Sub-Orch dropped new value Y (we missed the updated value)

Is there any reason you can't use a unique instance ID for the sub-orchestration. As you see from my code, I am using the item Id (which is my partition key in cosmos db) as the sub-orchestrator Id. Reason is that I am trying to achieve ordered handling per partition.

Update

It works if I don't await CallSubOrchestratorAsync but all partitions are blocking until the currently running sub-orch finishes

[cgillum]

I wonder if using durable entities might be a better option than sub-orchestrations. They may be better designed to handle this type of pattern: https://docs.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview#aggregator

[MhAllan]

I don't know @cgillum I am already over this problem I used different techniques but this needs fixing. thanks for suggesting =)