Terminating "completed" or non-existent durable function throws Grpc.Core.Exception

Description

Terminating durable function that has completed (RuntimeStatus=Failed | Completed | Terminated) or for non-existent instance id throws Grpc.Core.Exception exception with message Status(StatusCode="Unknown", Detail="Exception was thrown by handler.") For functions that are in running status, the termination works as expected and function is successfully terminated.

Expected behavior

Terminating completed or non-existent durable function will fail silently as documentation says.

Actual behavior

Terminating completed durable function or non-existent throws Grpc.Core.Exception exception with message Status(StatusCode="Unknown", Detail="Exception was thrown by handler.")

Relevant source code snippets

Basically try to create any function in the mentioned "completed" state and try to terminate it.

using System.Threading.Tasks;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.DurableTask;
using Microsoft.DurableTask.Client;

namespace Functions..TestTask;

public class Function
{
    [Function("TestTaskOrchestrator")]
    public async Task Orchestrator(
        [OrchestrationTrigger] TaskOrchestrationContext context)
    {
        await context.CallActivityAsync("TestTaskInitialize");
    }

    [Function("TestTaskInitialize")]
    public async Task Initialize([ActivityTrigger] object p)
    {
        //throw new Exception("test");
        await Task.Delay(10_000);
    }

    [Function( "Test_HttpTrigger")]
    public async Task<HttpResponseData> HttpTrigger(
        [HttpTrigger(AuthorizationLevel.Function, "get")] HttpRequestData request,
        [DurableClient] DurableTaskClient starter)
    {
        var options = new StartOrchestrationOptions { InstanceId = "testTaskInstanceId" };
        var intanceId = await starter.ScheduleNewOrchestrationInstanceAsync("TestTaskOrchestrator", options);
        return HttpResponseData.CreateResponse(request);
    }

    [Function("Test_HttpTrigger2")]
    public async Task<HttpResponseData> HttpTrigger2(
        [HttpTrigger(AuthorizationLevel.Function, "get")]
        HttpRequestData request,
        [DurableClient]
        DurableTaskClient starter)
    {
        await starter.TerminateInstanceAsync("testTaskInstanceId");
        return HttpResponseData.CreateResponse(request);
    }
}

Known workarounds

The most intuitive workaround is to get the instance, check if it is running and then call the terminationInstanceAsync method. Just a note that there is place for race condition, when the function will complete after GetInstanceAsync call, then the exception will be thrown anyway and we are not handling case with the non-existent function. This workaround is fragile but good enough for our use case for now.

        var taskInstance = await durableClient.GetInstanceAsync(instanceId);
        if (taskInstance.IsRunning)
        {
            await durableClient.TerminateInstanceAsync(instanceId, new TerminateInstanceOptions { Output = reason, Recursive = true });
        }

App Details

Durable Functions extension version (e.g. v1.8.3): 2.13.2, 2.13.4
Azure Functions runtime version (1.0 or 2.0): 2.0 (Isolated Worker)
Programming language used: C#

Azure / azure-functions-durable-extension