RetryOptions error-handler in Orchestration cannot capture right exception-type from Sub-Orchestration

[Nabakamal]

My question is related to 436 and 807- this is explained below:

Problem: Cannot get the right exception (via InnerException or FailureDetails) in the exception handler of RetryOptions (Handle)

Scenario: I have an Orchestration that calls a sub-orchestration using RetryOptions, like:

RetryOptions retryOptions = new RetryOptions(TimeSpan.FromMilliseconds(2500),3){
Handle = e =>
{
               /*
                    This exception never captures MyCustomException, which was originally thrown, 
                    either in FailureDetails or the InnerException property
              */

        SubOrchestrationFailedException tfe = e as SubOrchestrationFailedException;

        if (tfe != null && tfe.InnerException != null)
        {
            e = tfe.InnerException;
        }

        MyCustomException ce = e as MyCustomException; 
        if (ce != null)
        {
            LatestException = ce;  //LatestException is a variable of type MyCustomException
            return true;
        }
        return false;
    }};
    // IPayload is a custom type that is supposed to be returned from my sub-orchestration(if it ran successfully) 
    return await context.CreateSubOrchestrationInstanceWithRetry<IPayload>(typeof(FetchRatesSubOrchestration), retryOptions, 1);

The sub-orchestration's RunTask() looks like:

public override async Task<IPayload> RunTask(OrchestrationContext context, object input)
{
    List<AssetDTO> rates = new List<AssetDTO>();
    Reference1 reference1 = await context.ScheduleTask<Reference1>(typeof(TaskActivity1));
    await context.ScheduleTask<bool>(typeof(TaskActivity2), reference1);
    rates = await context.ScheduleTask<List<AssetDTO>>(typeof(TaskActivity3), reference1);
    List<string> datasReceived = rates.Select(x => x.TickerName).ToList();
    List<string> validDataPoints = _dbContext.SourceKeys.Select(t => t.SourceKeyValue).ToList();
    List<string> missingDataPoints = datasReceived.Except(validDataPoints).ToList();
    if (missingDataPoints.Count() > 0)
    {
        _logger.LogError($"Requested data points {string.Join(",", missingDataPoints)} were not returned in the response. Retrying.");
        throw new MyCustomException($"Requested data points {string.Join(",", missingDataPoints)} were not returned in the response. Retrying.");

        //OrchestrationFailureException innerExceptionToThrow = new OrchestrationFailureException($"Requested data points {string.Join(",", missingDataPoints)} were not returned in the response. Retrying.");
        //throw innerExceptionToThrow;
        // var exc =  new DurableTask.Core.Exceptions.SubOrchestrationFailedException($"Orchestration failed - {GetType().Name}", innerExceptionToThrow);
        // exc.FailureDetails
        // throw exc;
    }
    return new RatesAvailablePayload(rates);
}

The error thrown within the if-block in the sub-orchestration never bubbles-up to the Handle error-handler of the defined RetryOptions (within the parent orchestration).

Additionally, what else have I tried? a. Setting the ErrorPropagationMode to ErrorPropagationMode.UseFailureDetails (or ErrorPropagationMode.SerializeExceptions) doesn't help - the FailureDetails object of the exception and the InnerException property are always null.

b. I have tried debugging against the source of the framework, and, I believe to have seen the "Multiple ExecutionCompletedEvent found, potential corruption in state storage" message from the SerMarkerEvents() method in DurableTask.Core.OrchestrationRuntimeState class - but that was during one of the times I was debugging it.

c. While debugging the framework source, I noticed that the FailureDetails object is populated, for the most part, but due to the parallel-execution of the code it is a little difficult to debug as the control jumps from one class to the other, and then to the third - which makes following a trail difficult, at best

d. I have looked at most, if not all, of the tests in the DurableTask and the DurableTask-MSSQL repositories, but these haven't helped me in figuring out what I may still be missing

I'm using the following packages:

Microsoft.DurableTask.SqlServer - 1.1.1 Microsoft.Azure.DurableTask.Core - 2.10.0 (dependency brought in via Microsoft.DurableTask.SqlServer)

Please advise on how to get this resolved.

Thank you.

[Nabakamal]

@cgillum @jviau @papigers Can you please advise, when time permits? Thank you.