Orchestrator keeps triggering Activities despite having no Runtime Status

[flavio-vgt]

Description

We are using Durable Function Orchestrators to run data import jobs on our application. Since the start of the month we experienced a high increase in compute cost on our Azure bill. We found out that multiple Orchestrator Functions are calling multiple thousand Activity Functions each. The Orchestrator functions responsible did not have any Runtime Status set nor any CreatedTime. We are using the Application Routing architecture from here:

Expected behavior

Orchestrator Functions with no RuntimeStatus or CreatedTime should not be able to trigger Activity Functions.

Actual behavior

Over the span of a couple of days (~20) we found upwards of 100k Messages in the DurableTaskHistory table which linked to the Orchestrator function.

Relevant source code snippets

// insert code snippet here

Known workarounds

We looked into the Application Insight logs and manually deleted the Orchestrator Instance and the History from the Azure Table Storage, which killed the Orchestrator and prevented it from generating more traffic. (We left one Orchestrator Instance for analysis)

App Details

• Durable Functions extension version: v2.8.1:
• Azure Functions runtime version: v4:
• Programming language used: C#:

Screenshots

If deployed to Azure

• Timeframe issue observed: 20.10.2023-now
• Function App name: (see in Azure support request):
• Function name(s): FTPMeasurementService (orch), ProcessFTPFolderNew (activity)
• Azure region: Switzerland North
• Orchestration instance ID(s): service_b0395c9c-c6df-4220-993f-dab8cdfc9d9d_df9296dd-36f4-4b61-a9f1-1780ccb04083_9223370337672772991_f3303e19-4660-4bfc-8bee-3da7f95a38de
• Azure storage account name: see in Azure supoort request
• Azure Support Request id: TrackingID#2311200050005516

[ZhivomirZlatinov]

Hello team, I am the Azure support engineer currently working on case number 2311200050005516 related to this support request. I have reviewed the logs available on the App Service level, but could not find a cause for the issue. Would you be able to assist with this as we believe that there is a potential bug?

[cgillum]

Hi @flavio-vgt. Apologies for the delay. We're taking a look at this and will follow up with you via the support ticket.