System.OperationCanceledException: The operation was cancelled.

Report

There is a recurring stack trace that we have been seeing for a long time. At the same time we're not sure what to do about it.

Expected Behavior

The error explaining in human readable language what happened and what to do about it.

Actual Behavior

System.OperationCanceledException: The operation was cancelled.
 ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.
   at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.SendRequestAsync(IChannel channel, IHttpRequest request, Boolean expect100Continue, IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\DotNettyHttpBackend.cs:line 422
   at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.SendRequestAsync(IChannel channel, IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\DotNettyHttpBackend.cs:line 264
   at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\DotNettyHttpBackend.cs:line 139
   --- End of inner exception stack trace ---
   at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Http.Client.DotNetty\DotNettyHttpBackend.cs:line 174
   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable`1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\ChildPipeline.cs:line 38
   at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask`1 input) in D:\a\1\s\Proxy\Gateway.Pipeline\Extensions\ValueTaskExtensions.cs:line 28
   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.IO.CallServiceHandler.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Policies.General\IO\CallServiceHandler.cs:line 107
   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable`1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\ChildPipeline.cs:line 38
   at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask`1 input) in D:\a\1\s\Proxy\Gateway.Pipeline\Extensions\ValueTaskExtensions.cs:line 28
   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable`1 steps, CancellationToken cancellation) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineWalker.cs:line 67
   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.PipelineExecutor.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\a\1\s\Proxy\Gateway.Pipeline\PipelineExecutor.cs:line 215

Steps to Reproduce the Problem

Run the self hosted gateway
It shows up eventually

Logs from self-hosted gateway

{
    "id": "AgAAAZBu4fP-UuP_UwAAAAAAAAAYAAAAAEFaQnU0Z2Z5QUFBbGlmek11LW9landBdAAAACQAAAAAMDE5MDZlZTktZGI0NC00ZjY3LTk3M2ItN2EzOTBiODVkNzhm",
    "content": {
        "timestamp": "2024-07-01T15:20:20.222Z",
        "tags": [ "redacted"
        ],
        "host": "redacted",
        "service": "azure-api-gateway",
        "attributes": {
            "level": "Warning",
            "transportError": 0,
            "httpError": 0,
            "operationId": "redactedPOST",
            "source": "request-forwarder",
            "subscriptionId": "62f0fd9daa25a5722900a3e9",
            "httpMethod": "POST",
            "serviceName": "apim-gateway",
            "error": {
                "stack": "System.OperationCanceledException: The operation was cancelled.\n ---> System.Threading.Tasks.TaskCanceledException: A task was canceled.\n   at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.SendRequestAsync(IChannel channel, IHttpRequest request, Boolean expect100Continue, IPipelineContext context, CancellationToken cancellation) in D:\\a\\1\\s\\Proxy\\Gateway.Http.Client.DotNetty\\DotNettyHttpBackend.cs:line 422\n   at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.SendRequestAsync(IChannel channel, IPipelineContext context, CancellationToken cancellation) in D:\\a\\1\\s\\Proxy\\Gateway.Http.Client.DotNetty\\DotNettyHttpBackend.cs:line 264\n   at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\\a\\1\\s\\Proxy\\Gateway.Http.Client.DotNetty\\DotNettyHttpBackend.cs:line 139\n   --- End of inner exception stack trace ---\n   at Gateway.Http.Client.DotNetty.DotNettyHttpBackend.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\\a\\1\\s\\Proxy\\Gateway.Http.Client.DotNetty\\DotNettyHttpBackend.cs:line 174\n   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable`1 steps, CancellationToken cancellation) in D:\\a\\1\\s\\Proxy\\Gateway.Pipeline\\PipelineWalker.cs:line 67\n   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\\a\\1\\s\\Proxy\\Gateway.Pipeline\\ChildPipeline.cs:line 38\n   at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask`1 input) in D:\\a\\1\\s\\Proxy\\Gateway.Pipeline\\Extensions\\ValueTaskExtensions.cs:line 28\n   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.IO.CallServiceHandler.ProcessAsync(IPipelineContext context, CancellationToken cancellation) in D:\\a\\1\\s\\Proxy\\Gateway.Policies.General\\IO\\CallServiceHandler.cs:line 107\n   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable`1 steps, CancellationToken cancellation) in D:\\a\\1\\s\\Proxy\\Gateway.Pipeline\\PipelineWalker.cs:line 67\n   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.ChildPipeline.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\\a\\1\\s\\Proxy\\Gateway.Pipeline\\ChildPipeline.cs:line 38\n   at Gateway.Pipeline.Extensions.ValueTaskExtensions.Await[T](ValueTask`1 input) in D:\\a\\1\\s\\Proxy\\Gateway.Pipeline\\Extensions\\ValueTaskExtensions.cs:line 28\n   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.Policies.PipelineWalker.ExecuteAsync(IPipelineContext context, IEnumerable`1 steps, CancellationToken cancellation) in D:\\a\\1\\s\\Proxy\\Gateway.Pipeline\\PipelineWalker.cs:line 67\n   at Microsoft.WindowsAzure.ApiManagement.Proxy.Gateway.PipelineExecutor.ExecuteAsync(IPipelineContext context, CancellationToken cancellationToken) in D:\\a\\1\\s\\Proxy\\Gateway.Pipeline\\PipelineExecutor.cs:line 215",
                "kind": "System.OperationCanceledException",
                "message": "The operation was cancelled."
            },
            "apiId": "redacted-APIs",
            "timestamp": "2024-07-01T15:20:20.2222218Z",
            "tags": 208
        }
    }
}

Self-hosted Gateway Version

mcr.microsoft.com/azure-api-management/gateway:2.6.0

Deployment

Self-Managed (YAML/Helm)

Platform

Google Cloud

Kubernetes Version

1.26

Anything else?

No response

Azure / api-management-self-hosted-gateway

System.OperationCanceledException: The operation was cancelled. #290