Service bus message actions throwing Grpc.Core.RpcException when sending multiple requests

[MathBouma]

Description

This occurs with a C# isolated function, using .NET 8 and latest versions of all libraries.   For some reason, when using the message actions with batched service bus messages- some of them will fail with the following error: Call failed with gRPC error status. Status code: 'Unimplemented', Message: 'Service is unimplemented.

This looks similar to Azure/azure-functions-dotnet-worker#1974, however only occurs after multiple requests have been made within one function execution.

My best guess is some kind of limit, be it connections / pooling not being handled correctly.

Steps to reproduce

• Have a function app which receives a large batch of messages (experienced with ~100).
• Use message actions to deadletter, abandon or complete a large number of the messages.
• Have multiple function executions occurring in short time of each other.
  • Short cut could be to fill service bus with a decent number of messages, then call abandon on each message with a large retry set.

Some of the requests will then fail with the GRPC error.

My specific scenario was a function which:

• Receives a batch of messages.
• Tries to bulk insert messages into cosmos.
• Completes successful requests, and abandons failed requests
  • This is because comos can hit rate limits, and only allow a portion of the records to be created. Simply allowing the function to retry or complete would cause issues, as either requests would be duplicated or missed.

Happy to provide more info / source code as needed.

[kanetik]

Came looking for an answer on something very similar. We're getting bursts of RpcExceptions when we have a large number of concurrent messages we're processing. The error message isn't the same, but it's an RpcException on call to CompleteMessageAsync. We're also on .NET 8, isolated function, and latest version of all libraries (5.17.0 of Worker.Extentions.ServiceBus).

The message we get, though is Grpc.Core.RpcException: Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.. SocketException: An existing connection was forcibly closed by the remote host.", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request.")

[zmerdev]

Also seeing this in logs of a batched servicebus triggered function using message actions to manually complete/abandon/deadletter all messages. Our logging makes it difficult to trackdown where exactly the issue is cropping up.

.net 8 isolated service bus triggered batch azure function.

partial stack trace

Result: Failure
Exception: Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="Service is unimplemented.")

at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

[dmaples-midmark]

We have also seen this with low volumes of single messages. Running some tests yesterday (4/16), I saw 2 failures out of 20 test messages right after deploying a minor update in our development environment. Since then I've been unable to reproduce the exception intentionally after running several thousand messages through the function.

We just upgraded the function to .NET 8 Isolated and running the latest release version of all libraries:

Example Full Stack Trace:

Exception while executing function: Functions.ServiceContracts Result: Failure
Exception: Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="Service is unimplemented.")
   at Microsoft.Azure.Functions.Worker.ServiceBusMessageActions.CompleteMessageAsync(ServiceBusReceivedMessage message, CancellationToken cancellationToken) in D:\a\_work\1\s\extensions\Worker.Extensions.ServiceBus\src\ServiceBusMessageActions.cs:line 78
   at Datahub.MessageBroker.Function.Consumers.MidmarkServiceBusConsumerBase.HandleMessageExceptionAsync(Exception ex, ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions) in /home/vsts/work/1/s/self/Datahub.MessageBroker.Function/Consumers/MidmarkServiceBusConsumerBase.cs:line 182
   at Datahub.MessageBroker.Function.Consumers.MidmarkServiceBusConsumerBase.HandleIncomingServiceBusMessage(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions, String processName, String topicName, String subscriptionName) in /home/vsts/work/1/s/self/Datahub.MessageBroker.Function/Consumers/MidmarkServiceBusConsumerBase.cs:line 96
   at Datahub.MessageBroker.Function.Consumers.ASBConsumer.ServiceContractsFunctionAsync(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions) in /home/vsts/work/1/s/self/Datahub.MessageBroker.Function/Consumers/ASBConsumer.cs:line 34
   at Datahub.MessageBroker.Function.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in /home/vsts/work/1/s/self/Datahub.MessageBroker.Function/Microsoft.Azure.Functions.Worker.Sdk.Generators/Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator/GeneratedFunctionExecutor.g.cs:line 38
   at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
   at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack:    at Microsoft.Azure.Functions.Worker.ServiceBusMessageActions.CompleteMessageAsync(ServiceBusReceivedMessage message, CancellationToken cancellationToken) in D:\a\_work\1\s\extensions\Worker.Extensions.ServiceBus\src\ServiceBusMessageActions.cs:line 78
   at Datahub.MessageBroker.Function.Consumers.MidmarkServiceBusConsumerBase.HandleMessageExceptionAsync(Exception ex, ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions) in /home/vsts/work/1/s/self/Datahub.MessageBroker.Function/Consumers/MidmarkServiceBusConsumerBase.cs:line 182
   at Datahub.MessageBroker.Function.Consumers.MidmarkServiceBusConsumerBase.HandleIncomingServiceBusMessage(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions, String processName, String topicName, String subscriptionName) in /home/vsts/work/1/s/self/Datahub.MessageBroker.Function/Consumers/MidmarkServiceBusConsumerBase.cs:line 96
   at Datahub.MessageBroker.Function.Consumers.ASBConsumer.ServiceContractsFunctionAsync(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions) in /home/vsts/work/1/s/self/Datahub.MessageBroker.Function/Consumers/ASBConsumer.cs:line 34
   at Datahub.MessageBroker.Function.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in /home/vsts/work/1/s/self/Datahub.MessageBroker.Function/Microsoft.Azure.Functions.Worker.Sdk.Generators/Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator/GeneratedFunctionExecutor.g.cs:line 38
   at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
   at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

[dmaples-midmark]

There seems to be a possible correlation with the RpcExceptions and our function initializing. Since we are in consumption plan, our function is often not running during periods of downtime. The following telemetry is a 6 hour snapshot from Azure showing RpcExceptions overlayed with Process CPU.

What is interesting here is that all of the exceptions overlap with periods where the function was not reporting Process CPU yet, likely because it was initializing. There may be some time variation between the reporting of these individual metrics that could invalidate this. However, I thought it might be significant enough to share here.

[janus007]

I'm experiencing the exact same behavior and as this wasn't enough I get a ton of object disposed.

These Object Disposed are apparently an issue that can't be fixed. I have spent tremendous amounts of time, trying to understand the problem. Others have same issues: https://github.com/Azure/azure-sdk-for-net/issues/19731

If I use the exact same SB Queue, but instead switch to a Topic everything works fine. What is going on?

It seems very unprofessional from you .NET guys, that the .NET 7.0 Isolated stops support May 10 2024, and we are left with this major bug in .NET 8 Isolated.

I can't imagine how much money there is wasted in the World due to this.

System.ObjectDisposedException: Cannot access a disposed object. Object name: 'System.Net.WebSockets.ClientWebSocket'. at System.Net.WebSockets.ClientWebSocket.get_ConnectedWebSocket() at System.Net.WebSockets.ClientWebSocket.SendAsync(ArraySegment1 buffer, WebSocketMessageType messageType, Boolean endOfMessage, CancellationToken cancellationToken) at Microsoft.Azure.Amqp.Transport.WebSocketTransport.WriteAsync(TransportAsyncCallbackArgs args) at Microsoft.Azure.Amqp.AsyncIO.AsyncBufferWriter.Write(TransportAsyncCallbackArgs args) --- End of stack trace from previous location --- at Microsoft.Azure.Amqp.AsyncResult.End[TAsyncResult](IAsyncResult result) at Microsoft.Azure.Amqp.Transport.AmqpTransportInitiator.<>c.b__17_1(IAsyncResult r) at System.Threading.Tasks.TaskFactory1.FromAsyncCoreLogic(IAsyncResult iar, Func2 endFunction, Action1 endAction, Task1 promise, Boolean requiresSynchronization) --- End of stack trace from previous location --- at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.CreateAndOpenConnectionAsync(Version amqpVersion, Uri serviceEndpoint, Uri connectionEndpoint, ServiceBusTransportType transportType, IWebProxy proxy, String scopeIdentifier, TimeSpan timeout) at Microsoft.Azure.Amqp.FaultTolerantAmqpObject1.OnCreateAsync(TimeSpan timeout, CancellationToken cancellationToken) at Microsoft.Azure.Amqp.Singleton1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken) at Microsoft.Azure.Amqp.Singleton1.GetOrCreateAsync(TimeSpan timeout, CancellationToken cancellationToken) at Azure.Messaging.ServiceBus.Amqp.AmqpConnectionScope.OpenReceiverLinkAsync(String identifier, String entityPath, TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String sessionId, Boolean isSessionReceiver, CancellationToken cancellationToken) at Azure.Messaging.ServiceBus.Amqp.AmqpReceiver.OpenReceiverLinkAsync(TimeSpan timeout, UInt32 prefetchCount, ServiceBusReceiveMode receiveMode, String identifier, CancellationToken cancellationToken)

[russaram-bham]

We are also experiencing the same issue when executing an Azure Function via a Service Bus Trigger (.Net 8 Isolated) when Cosmos returns a number of 429s.

Messages are not placed on the DLQ but a RpcExceptions is thrown.

[nkalfov]

I am also experiencing this issue. I have an Azure Function writing messages in an Azure Service Bus Queue and another Azure Function is triggered by those messages. The exception is thrown on both sides, but most often happens when executing: await messageActions.CompleteMessageAsync(message);

Both Functions are written in .NET 8 and run in an Azure Function App in isolated mode. Both functions connect to Azure Service Bus with AMPQ over TCP. Both functions depend on packages:

<FrameworkReference Include="Microsoft.AspNetCore.App" />
<PackageReference Include="Azure.Messaging.ServiceBus" Version="7.17.4" />
<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.21.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http" Version="3.1.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore" Version="1.2.1" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.17.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.Timer" Version="4.3.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.1" />
<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" />
<PackageReference Include="Microsoft.Extensions.Azure" Version="1.7.2" />

I do see it occasionally in Application Insights. What is the reason? What is the resolution?

Stack Trace:

Result: Failure
Exception: Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="Service is unimplemented.")
at Microsoft.Azure.Functions.Worker.ServiceBusMessageActions.CompleteMessageAsync(ServiceBusReceivedMessage message, CancellationToken cancellationToken) in D:\a\_work\1\s\extensions\Worker.Extensions.ServiceBus\src\ServiceBusMessageActions.cs:line 78
at *******.Run(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions) in /opt/atlassian/pipelines/agent/build/src/********/*******.cs:line 47
at *******.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in /opt/atlassian/pipelines/agent/build/src/*******/Microsoft.Azure.Functions.Worker.Sdk.Generators/Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator/GeneratedFunctionExecutor.g.cs:line 42
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack: at Microsoft.Azure.Functions.Worker.ServiceBusMessageActions.CompleteMessageAsync(ServiceBusReceivedMessage message, CancellationToken cancellationToken) in D:\a\_work\1\s\extensions\Worker.Extensions.ServiceBus\src\ServiceBusMessageActions.cs:line 78
at *******.Run(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions) in /opt/atlassian/pipelines/agent/build/src/*******/*******.cs:line 47
at*******.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in /opt/atlassian/pipelines/agent/build/src/*******/Microsoft.Azure.Functions.Worker.Sdk.Generators/Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator/GeneratedFunctionExecutor.g.cs:line 42
at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

[Vixan]

As others have mentioned, I'm also encountering this issue for a couple of weeks. Whether I'm using CompleteMessageAsync, DeadLetterMessageAsync or letting the function autocomplete messages without using ServiceBusMessageActions, I'm encountering these RpcExceptions with status "Unimplemented" or "Unavailable".

Due to the fact that my functions create side effects (update database records, file creation, etc.), not being able to rely on message completion/dead-lettering after processing them causes multiple issues. I'd appreciate any help 🙂.

[wtrombly]

As others have mentioned I am also experiencing this issue when using CompleteMessageAsync and DeadLetterMessageAsync. We are using azure serverless functions with a consumption plan on .NET 8. Would appreciate any help.

[AlexEngblom]

We are also experiencing this issue after upgrading to NET8 and isolated Function base.

[MichalDulski]

We're experiencing a very similar issue too. Our Azure Function uses .NET 8 isolated-worker model and runs on a consumption plan. We do not batch messages. The function works fine when the load is low (let's say 20 messages per minute) , but when the function receives more messages than that, it sometimes fails due to the Grpc.Core.RpcException being thrown by the CompleteMessageAsync or DeferMessageAsync methods. Our function processes over 3000 messages daily, and we encounter this issue once or twice almost every day. Although it’s not consistent, there are times when everything works properly, which might be related to slightly lower load on some days.

Stack trace:

Result: Failure
Exception: Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="Service is unimplemented.")
   at Microsoft.Azure.Functions.Worker.ServiceBusMes`sageActions.CompleteMessageAsync(ServiceBusReceivedMessage message, CancellationToken cancellationToken) in D:\a\_work\1\s\extensions\Worker.Extensions.ServiceBus\src\ServiceBusMessageActions.cs:line 78
   at *.*.*.Run(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions, CancellationToken cancellationToken) in Z:\*\*\*\FUNCTION_FILE.cs:line 87
   at *.*.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in Z:\*\*\*\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 38
   at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
   at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88
Stack:    at Microsoft.Azure.Functions.Worker.ServiceBusMessageActions.CompleteMessageAsync(ServiceBusReceivedMessage message, CancellationToken cancellationToken) in D:\a\_work\1\s\extensions\Worker.Extensions.ServiceBus\src\ServiceBusMessageActions.cs:line 78
   at *.*.*.Run(ServiceBusReceivedMessage message, ServiceBusMessageActions messageActions, CancellationToken cancellationToken) in Z:\*\*\*\FUNCTION_FILE.cs:line 87
   at *.*.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in Z:\*\*\*\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 38
   at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
   at Microsoft.Azure.Functions.Worker.Extensions.Http.AspNetCore.FunctionsHttpProxyingMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\extensions\Worker.Extensions.Http.AspNetCore\src\FunctionsMiddleware\FunctionsHttpProxyingMiddleware.cs:line 34
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 77
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

I would appreciate any help. If there's something more I could do to help, please let me know.

[RubenDelange]

We are also experiencing this issue after upgrading to NET8 and using isolated Functions. Also using batched mode and message actions to deadletter, abandon, or complete a message.

Exception message: One or more errors occurred. (Status(StatusCode="Unimplemented", Detail="Service is unimplemented."))

Stack trace from app insights:

   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionInvoker`2.<>c.<InvokeAsync>b__6_0(Task`1 t) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionInvoker.cs:line 32
   at System.Threading.Tasks.ContinuationResultTaskFromResultTask`2.InnerInvoke()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
--- End of stack trace from previous location ---
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)
--- End of stack trace from previous location ---
   at Microsoft.Azure.Functions.Worker.Invocation.DefaultFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\Invocation\DefaultFunctionExecutor.cs:line 49
   at ****.Function.DirectFunctionExecutor.ExecuteAsync(FunctionContext context) in D:\a\1\s\src\****\****.Function\src\****.Function\obj\Release\net8.0\Microsoft.Azure.Functions.Worker.Sdk.Generators\Microsoft.Azure.Functions.Worker.Sdk.Generators.FunctionExecutorGenerator\GeneratedFunctionExecutor.g.cs:line 49
   at Microsoft.Azure.Functions.Worker.OutputBindings.OutputBindingsMiddleware.Invoke(FunctionContext context, FunctionExecutionDelegate next) in D:\a\_work\1\s\src\DotNetWorker.Core\OutputBindings\OutputBindingsMiddleware.cs:line 13
   at Microsoft.Azure.Functions.Worker.FunctionsApplication.InvokeFunctionAsync(FunctionContext context) in D:\a\_work\1\s\src\DotNetWorker.Core\FunctionsApplication.cs:line 89
   at Microsoft.Azure.Functions.Worker.Handlers.InvocationHandler.InvokeAsync(InvocationRequest request) in D:\a\_work\1\s\src\DotNetWorker.Grpc\Handlers\InvocationHandler.cs:line 88

Packages:

<FrameworkReference Include="Microsoft.AspNetCore.App" />
<PackageReference Include="Azure.Messaging.ServiceBus" Version="7.17.5" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.ApplicationInsights" Version="1.2.0" />
<PackageReference Include="Microsoft.ApplicationInsights.WorkerService" Version="2.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.22.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.ServiceBus" Version="5.19.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.17.2" />

Errors regularly occurring in App Insights:

[fabiocav]

Apologies for the delayed response here.

Thank you for the information shared so far, this gives us a good idea of what the potential issue is and will be investigating and posting updates here as we make progress.

This is looking like a host problem, so we'll be transferring this issue to that repo.

[Kalyan-Ad-Shell]

I am also experiencing 2 similar issues related to grpc:

Error 1:

Result: Cancelled Exception: Grpc.Core.RpcException: Status(StatusCode="Cancelled", Detail="Call canceled by the client.", DebugException="System.OperationCanceledException: The operation was canceled.") ---> System.OperationCanceledException: The operation was canceled. --- End of inner exception stack trace --- at Microsoft.Azure.Functions.Worker.ServiceBusMessageActions.DeadLetterMessageAsync(ServiceBusReceivedMessage message, Dictionary`2 propertiesToModify, String deadLetterReason, String deadLetterErrorDescription, CancellationToken cancellationToken)

Error 2:

Result: Failure Exception: Grpc.Core.RpcException: Status(StatusCode="Unimplemented", Detail="Service is unimplemented.") at Microsoft.Azure.Functions.Worker.ServiceBusMessageActions.DeadLetterMessageAsync(ServiceBusReceivedMessage message, Dictionary`2 propertiesToModify, String deadLetterReason, String deadLetterErrorDescription, CancellationToken cancellationToken)