Azure / durabletask

Durable Task Framework allows users to write long running persistent workflows in C# using the async/await capabilities.
Apache License 2.0
1.53k stars 296 forks source link

GetOrchestrationStateAsync hangs infinitely #975

Closed twosatnams closed 1 year ago

twosatnams commented 1 year ago

Initialization

// Durable Task Framework with Azure Queues
AzureStorageOrchestrationServiceSettings settings = new AzureStorageOrchestrationServiceSettings
{
    StorageAccountDetails = new StorageAccountDetails
    {
        ConnectionString = environment.AzureStorageConnectionString
    },
    TaskHubName = $"{currentEnvironment.ToString().ToLowerInvariant()}tasks",
    LoggerFactory = loggerFactory
};
AzureStorageOrchestrationService orchestrationService = new AzureStorageOrchestrationService(settings);
builder.Services.AddSingleton(new TaskHubClient(orchestrationService, loggerFactory: loggerFactory));

Calling Code

    // Doesn't work right now
    // Hang infinitely
    public bool IsAlreadyEnqueued(string instanceId)
    {
        _logger.LogInformation("Checking if {} is already enqueued", instanceId);
        bool state = _taskHubClient.GetOrchestrationStateAsync(instanceId, false).Result != null;
        _logger.LogInformation("{} is already enqueued?: {}", instanceId, state);
        return state;
    }

Expected Behavior

The call should return a list of orchestration states for the most current execution.

Actual Behavior

The function never returns. I see the log statement before GetOrchestrationStateAsync gets printed. And another log statement that DTF prints. But the result is never returned, and then it waits infinitely. This behavior remains the same regardless of whether I try with an instanceId that I know has enqueued in the past or a new random one that I know has never been scheduled.

Checking if cb592395-d268-4bde-b020-3414b3941c0f is already enqueued
2023-39-23T07:39:01.122198Z info: DurableTask.Core[44] Fetching tracking state for instance 'cb592395-d268-4bde-b020-3414b3941c0f'

Environment

OS: macOS Big Sur DTF Version: 2.14.0 Persistence: AzureStorage

Other Notes

CreateOrchestrationInstanceAsync works fine, so it's probably not an access issue local to my setup

cgillum commented 1 year ago

The problem might be related to your use of .Result for this async API in your calling code. In certain environments, calling .Result on a task can result in deadlocks. This is more a .NET thing and not so much a behavior of this particular API.

Try instead rewriting it to look like this:

public async Task<bool> IsAlreadyEnqueuedAsync(string instanceId)
{
    _logger.LogInformation("Checking if {} is already enqueued", instanceId);
    var result = await _taskHubClient.GetOrchestrationStateAsync(instanceId, false);
    bool state = result != null;
    _logger.LogInformation("{} is already enqueued?: {}", instanceId, state);
    return state;
}
twosatnams commented 1 year ago

Thanks @cgillum, I tried that solution you suggested and it all works as expected now. Closing