search

Azure / durabletask

Durable Task Framework allows users to write long running persistent workflows in C# using the async/await capabilities.
Apache License 2.0
1.51k stars 289 forks source link

System.NullReferenceException in AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync #487

Open thirus opened 3 years ago

thirus commented 3 years ago

Version: Core 2.4.1, AzureStorate: 1.8.1

For our use case we use sub orchestrations and external events, however we observed random NPE and our main orchestration instance get stuck since the sub orchestration instance never gets created.

Exception:

DurableTask.Core[29]
      TaskOrchestrationDispatcher-1bc6efed64814fb897c9e58d6f22c01b-0: Unhandled exception with work item 'fb874f1e8645614ea67e2330b4c60fcd': System.NullReferenceException: Object reference not set to an instance of an object.
         at DurableTask.AzureStorage.AzureStorageOrchestrationService.CompleteTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkItem workItem, OrchestrationRuntimeState newOrchestrationRuntimeState, IList`1 outboundMessages, IList`1 orchestratorMessages, IList`1 timerMessages, TaskMessage continuedAsNewMessage, OrchestrationState orchestrationState) in C:\source\durabletask\src\DurableTask.AzureStorage\AzureStorageOrchestrationService.cs:line 1005
         at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) in C:\source\durabletask\src\DurableTask.Core\TaskOrchestrationDispatcher.cs:line 463
         at DurableTask.Core.TaskOrchestrationDispatcher.OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) in C:\source\durabletask\src\DurableTask.Core\TaskOrchestrationDispatcher.cs:line 194
         at DurableTask.Core.WorkItemDispatcher`1.ProcessWorkItemAsync(WorkItemDispatcherContext context, Object workItemObj) in C:\source\durabletask\src\DurableTask.Core\WorkItemDispatcher.cs:line 459

      Backing off for 10 seconds until 5 successful operations

While debugging locally we found this line gets runtimeState.OrchestrationInstance as null. https://github.com/Azure/durabletask/blob/bcc97e874baba09c722a83c164bb3ab2245017ec/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs#L948

below logic in the same method assumes the runtimeState.OrchestrationInstance can be null but not that line. example line that expects nullable instance https://github.com/Azure/durabletask/blob/bcc97e874baba09c722a83c164bb3ab2245017ec/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs#L972

thirus commented 3 years ago

We also could not able to find any records in the azure table storage for the sub orchestration but the main and other sub orch created by the main were available.

Just to clarify the same orchestration code (no changes anywhere in the codebase) with same input works most of time.