Failure details from sub-orchestrators come back as '(unknown)'

[PaulBDavies]

Description

When a sub-orchestration fails, it throws a 'TaskFailedException' in the parent orchestrator. However when I catch this error, the FailureDetails come out as '(unknown)' for both ErrorType and ErrorMessage. I'm assuming this isn't intended behaviour, since when I try the same thing with activity functions I'm able to retrieve useful information from these fields.

Expected behavior

The FailureDetails of the TaskFailedException contains the error message/type of the exception that caused the failure.

Actual behavior

The FailureDetails of the TaskFailedException both contain the string '(unknown)'

Relevant source code snippets

Here is some pseudocode similar to my own, I'm performing the same operation for multiple items in a list. If one fails I still want to perform the operation for the remainder of the items in the list, but I want to keep track of any errors that happened along the way.

foreach(var item in list)
{
    try
    {
        await context.CallSubOrchestratorAsync(nameof(FailingFunction), item);
    }
    catch(TaskFailedException ex)
    {
        var error = ex.FailureDetails?.ErrorMessage ?? "Unknown Error";
        errors.Add(error);
    }
}
// I'd expect 'errors' here to contain a list of error messages, which I can log/throw as an aggregate exception
// or something, but they are all just '(unknown)'

Known workarounds

N/A

App Details

• Durable Functions extension version: v1.0.0 (although I did try upgrading to v1.1.0 and saw the same issue) - This is the package version on nuget and doesn't seem to line up with the tags on this repo?
• Azure Functions runtime version: v4
• Programming language used: C#

Screenshots

N/A

If deployed to Azure

N/A

[JohanKlijn]

Any news on this issue?

We use a lot of sub-orchestrators and our main orchestrator is handling/logging all the exceptions and now we all the relevant information for debugging issues are not available any more.

[cgillum]

@JohanKlijn a new set of nuget packages were released yesterday and have some fixes related to exception propagation. Can you try upgrading to the latest to see if this issue is resolved for you?

[PaulBDavies]

This seems to have resolved the issue for me

[JohanKlijn]

Unfortunately I isn't fixed for me. Let me explain why....

As far as I can remember it was decided to not propagate exceptions any more, but use the FailureDetails on the TaskFailedException instead, right? (I am not 100% sure about this).

When I look at the current implementation is unclear what we should check, because:

• the TaskFailedException can have a InnerException
• the TaskFailedException can have FailureDetails with an InnerFailure.

In my example we have the following 'flow': Main Orchestration -> Sub Orchestrator -> Activity

The Activity is throwing an exception like: throw new InvalidOperationException("This is an exception from an activity in a sub-orchestrator.");

I the main orchestrator I would expect a TaskFailedException with the following structure:

1. A TaskFailedException with no InnerException
2. TaskFailedException with a FailureDetails with the following structure:
   • TaskFailedException.FailureDetails.ErrorType => "SubOrchestrationFailedException"
   • TaskFailedException.FailureDetails.ErrorMessage => "Task 'SubOrchestrator' (#0) failed with an unhandled exception" (or something like this)
   • TaskFailedException.FailureDetails.InnerFailure.ErrorType => "InvalidOperationException"
   • TaskFailedException.FailureDetails.InnerFailure.ErrorMessage => "This is an exception from an activity in a sub-orchestrator."

But instead a TaskFailedException with the following structure returned:

1. A TaskFailedException with InnerException of type SubOrchestrationFailedException with the following structure:
   • TaskFailedException.FailureDetails.ErrorType => "Microsoft.DurableTask.TaskFailedException"
   • TaskFailedException.FailureDetails.ErrorMessage => "Task 'ThrowExceptionActivity' (#0) failed with an unhandled exception: This is an exception from an activity in a sub-orchestrator."
2. The SubOrchestrationFailedException container a FailureDetails with the following structure:
   • TaskFailedException.FailureDetails.ErrorType => "Microsoft.DurableTask.TaskFailedException"
   • TaskFailedException.FailureDetails.ErrorMessage => "Task 'ThrowExceptionActivity' (#0) failed with an unhandled exception: This is an exception from an activity in a sub-orchestrator."
   • TaskFailedException.FailureDetails.InnerFailure.ErrorType => "DurableTask.Core.Exceptions.TaskFailedException"
   • TaskFailedException.FailureDetails.InnerFailure.ErrorMessage => "Task 'ThrowExceptionActivity' (#0) failed with an unhandled exception: Exception of type 'DurableTask.Core.Exceptions.TaskFailedException' was thrown."

This is the code I use:

using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;
using Microsoft.Extensions.Logging;

namespace FunctionApp10
{
    public static class Function1
    {
        [Function("Function1_HttpStart")]
        public static async Task<HttpResponseData> HttpStart(
           [HttpTrigger(AuthorizationLevel.Anonymous, "get", "post")] HttpRequestData req,
           [DurableClient] DurableTaskClient client,
           FunctionContext executionContext)
        {

            // Function input comes from the request content.
            string instanceId = await client.ScheduleNewOrchestrationInstanceAsync(
                nameof(Function1));

            return client.CreateCheckStatusResponse(req, instanceId);
        }

        [Function(nameof(Function1))]
        public static async Task<List<string>> RunOrchestrator(
            [OrchestrationTrigger] TaskOrchestrationContext context)
        {
            ILogger logger = context.CreateReplaySafeLogger(nameof(Function1));
            logger.LogInformation("Saying hello.");
            var outputs = new List<string>();

            // Handle exception when calling an sub orchestaror
            try
            {
                await context.CallSubOrchestratorAsync<string>("SubOrchestrator").ConfigureAwait(true);
            }

            catch (Microsoft.DurableTask.TaskFailedException exception)
            {
                logger.LogInformation("I would expect:");
                logger.LogInformation("exception.FailureDetails.ErrorType has excpected value: {IsExcpectedValue}", exception.FailureDetails.ErrorType == "SubOrchestrationFailedException");
                logger.LogInformation("exception.FailureDetails.ErrorMessage has excpected value: {IsExcpectedValue}", exception.FailureDetails.ErrorMessage == "Task 'SubOrchestrator' (#0) failed with an unhandled exception");
                logger.LogInformation("exception.FailureDetails.InnerFailure.ErrorType has excpected value: {IsExcpectedValue}", exception.FailureDetails.InnerFailure != null && exception.FailureDetails.InnerFailure.ErrorType == "InvalidOperationException");
                logger.LogInformation("exception.FailureDetails.InnerFailure.ErrorMessage has excpected value: {IsExcpectedValue}", exception.FailureDetails.InnerFailure != null && exception.FailureDetails.InnerFailure.ErrorMessage == "This is an exception from an activity in a sub-orchestrator.");
            }

            return outputs;
        }

        [Function("SubOrchestrator")]
        public static async Task<string> RunAsync([OrchestrationTrigger] TaskOrchestrationContext context)
        {
            await context.CallActivityAsync<string>(nameof(Function1.ThrowExceptionActivity));

            return string.Empty;
        }

        [Function(nameof(ThrowExceptionActivity))]
        public static string ThrowExceptionActivity([ActivityTrigger] string name, FunctionContext executionContext)
        {
            throw new InvalidOperationException("This is an exception from an activity in a sub-orchestrator.");

        }
    }
}

By the way I am running Azure Functions in the isolated worker model.

[andrew-xu-asurion]

we're having the same issue, the details of exception thrown at sub orchestration cannot be captured at orchestration. Are there any solutions?

[asos-gurpreetsingh]

Having same issues

[cgillum]

@asos-gurpreetsingh can you confirm what version of the Microsoft.Azure.Functions.Worker.Extensions.DurableTask nuget package you're using?

[JohanKlijn]

@cgillum : for me it is also still not working, see my comment on april, https://github.com/Azure/azure-functions-durable-extension/issues/2689#issuecomment-2033829636

[cgillum]

Thanks @JohanKlijn for the detailed analysis and repro steps. We'll take a look at this.

[asos-gurpreetsingh]

1.1.7 @cgillum