search

dapr / components-contrib

Community driven, reusable components for distributed apps
Apache License 2.0
548 stars 480 forks source link

Dapr Sidecar fails to publish message to azure service bus #3559

Open MattCosturos opened 1 month ago

MattCosturos commented 1 month ago

Expected Behavior

Dapr service bus component should be able to publish messages to the azure service bus.

Actual Behavior

After a period of running, the Dapr Sidecar contains logs indicating a failure.

time="2024-10-01T20:37:57.486086085Z" level=error msg="Too many failed attempts while publishing Service Bus message (nil): context canceled" app_id=actor-microservice component="status-pubsub (pubsub.azure.servicebus/v1)" instance=actor-microservice-8649476968-28bzt scope=dapr.contrib type=log ver=1.13.5
time="2024-10-01T20:37:59.447828639Z" level=error msg="Too many failed attempts while publishing Service Bus message (nil): context canceled" app_id=actor-microservice component="status-pubsub (pubsub.azure.servicebus/v1)" instance=actor-microservice-8649476968-28bzt scope=dapr.contrib type=log ver=1.13.5
time="2024-10-01T20:38:02.723978748Z" level=error msg="Too many failed attempts while publishing Service Bus message (nil): context canceled" app_id=actor-microservice component="status-pubsub (pubsub.azure.servicebus/v1)" instance=actor-microservice-8649476968-28bzt scope=dapr.contrib type=log ver=1.13.5

Using Dapr 1.13.5 in AKS.

Steps to Reproduce the Problem

Unable to repo on demand. Pods run for a while then errors appear.

Possibly related to #2617 ?

yaron2 commented 1 month ago

Can you try this with Dapr 1.14.4? The SDK for ASB was updated and it includes some fixes that may be relevant to what you're seeing.

MattCosturos commented 1 month ago

Yes I will update to 1.14.4 and test that out.

MattCosturos commented 1 month ago

Now running 1.14.4. Haven't been running long enough to be confident the Service bus error is gone, will keep monitoring.

I am now seeing this new type of message in the logs. level=warning msg="Error processing operation DaprBuiltInActorRetries. Retrying in 1s…" app_id=actor-microservice instance=actor-microservice-7cc7fb6b4b-hjs76 scope=dapr.runtime type=log ver=1.14.4

Is there any way to figure out what operation failed from the daprd logs?

MattCosturos commented 1 month ago

time="2024-10-11T14:58:19.041817468Z" level=error msg="Too many failed attempts while publishing Service Bus message (nil): context canceled" app_id=actor-microservice component="status-pubsub (pubsub.azure.servicebus/v1)" instance=actor-microservice-54bb48b7bc-cz5pr scope=dapr.contrib type=log ver=1.14.4

Still seeing this error with 1.14.4

MattCosturos commented 1 month ago

Looking at our logs in more detail, from our code, we catch the timeout ex. We then build a new dapr client and try again. We will try 3 times before giving up. The curious thing is that our logs all have the same timestamp. I assumed context cancelled meant timeout, but with all 3 attempts having the same timestamp, I feel like there couldn't have been a timeout since it didn't wait any time.

I am not sure, but I think the next call we make to publish the next message to the service bus succeeds. I think this since our logs are not full of errors publishing to the service bus.

daprd error

2024-10-11T10:58:19-04:00 time="2024-10-11T14:58:19.041817468Z" level=error msg="Too many failed attempts while publishing Service Bus message (nil): context canceled" app_id=actor-microservice component="status-pubsub (pubsub.azure.servicebus/v1)" instance=actor-microservice-54bb48b7bc-cz5pr scope=dapr.contrib type=log ver=1.14.4

C# logs.


2024-10-11T10:58:19-04:00 fail: MyNamespace.Proxies.StatusChange.MessagePublishProxy[0]
2024-10-11T10:58:19-04:00       Exception in PublishMessage.
2024-10-11T10:58:19-04:00       System.OperationCanceledException: The operation was canceled.
2024-10-11T10:58:19-04:00        ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
2024-10-11T10:58:19-04:00          at System.Net.Http.Http2Connection.Http2Stream.<>c.<WaitForDataAsync>b__88_0(Object s, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.CancelCall(Status status)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.CancelCallFromCancellationToken(CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.<>c.<RegisterCancellation>b__70_0(Object state, CancellationToken ct)
2024-10-11T10:58:19-04:00          at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
2024-10-11T10:58:19-04:00          at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
2024-10-11T10:58:19-04:00          at System.Threading.TimerQueue.FireNextTimers()
2024-10-11T10:58:19-04:00          at System.Threading.ThreadPoolWorkQueue.Dispatch()
2024-10-11T10:58:19-04:00          at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
2024-10-11T10:58:19-04:00       --- End of stack trace from previous location ---
2024-10-11T10:58:19-04:00          at System.Net.Http.Http2Connection.Http2Stream.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-10-11T10:58:19-04:00          at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
2024-10-11T10:58:19-04:00          --- End of inner exception stack trace ---
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.ResolveException(String summary, Exception ex, Nullable`1& status, Exception& resolvedException)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Balancer.Internal.BalancerHttpHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
2024-10-11T10:58:19-04:00          at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
2024-10-11T10:58:19-04:00          at System.Net.Http.DiagnosticsHandler.SendAsyncCore(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
2024-10-11T10:58:19-04:00          at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.TrySetCanceled(CancellationToken tokenToRecord, Object cancellationException)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
2024-10-11T10:58:19-04:00          at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.SetException(Exception exception, Task`1& taskField)
2024-10-11T10:58:19-04:00          at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunFromThreadPoolDispatchLoop(Thread threadPoolThread, ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
2024-10-11T10:58:19-04:00          at System.Threading.ThreadPoolWorkQueue.Dispatch()
2024-10-11T10:58:19-04:00          at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
2024-10-11T10:58:19-04:00       --- End of stack trace from previous location ---
2024-10-11T10:58:19-04:00          at Dapr.Client.DaprClientGrpc.MakePublishRequest(String pubsubName, String topicName, ByteString content, Dictionary`2 metadata, String dataContentType, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at MyNamespace.Proxies.StatusChange.MessagePublishProxy.PublishMessage(Object message, String pubsubName, String topicName) in /src/Shared/MyNamespace/Proxies/MessagePublish/MessagePublishProxy.cs:line 109
2024-10-11T10:58:19-04:00 warn: MyNamespace.Proxies.StatusChange.MessagePublishProxy[0]
2024-10-11T10:58:19-04:00       Disposing DaprClient.
2024-10-11T10:58:19-04:00 warn: MyNamespace.Proxies.StatusChange.MessagePublishProxy[0]
2024-10-11T10:58:19-04:00       Building new DaprClient.
2024-10-11T10:58:19-04:00 fail: MyNamespace.Proxies.StatusChange.MessagePublishProxy[0]
2024-10-11T10:58:19-04:00       Exception in PublishMessage.
2024-10-11T10:58:19-04:00       System.OperationCanceledException: The operation was canceled.
2024-10-11T10:58:19-04:00        ---> System.OperationCanceledException: The operation was canceled.
2024-10-11T10:58:19-04:00          at System.Threading.CancellationToken.ThrowOperationCanceledException()
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
2024-10-11T10:58:19-04:00          --- End of inner exception stack trace ---
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.ResolveException(String summary, Exception ex, Nullable`1& status, Exception& resolvedException)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.StartUnaryCore(HttpContent content)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.HttpClientCallInvoker.AsyncUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
2024-10-11T10:58:19-04:00          at Grpc.Core.Interceptors.InterceptingCallInvoker.<AsyncUnaryCall>b__4_0[TRequest,TResponse](TRequest req, ClientInterceptorContext`2 ctx)
2024-10-11T10:58:19-04:00          at Grpc.Core.Interceptors.InterceptingCallInvoker.AsyncUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
2024-10-11T10:58:19-04:00          at Dapr.Client.DaprClientGrpc.MakePublishRequest(String pubsubName, String topicName, ByteString content, Dictionary`2 metadata, String dataContentType, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
2024-10-11T10:58:19-04:00          at Dapr.Client.DaprClientGrpc.PublishEventAsync[TData](String pubsubName, String topicName, TData data, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at MyNamespace.Proxies.StatusChange.MessagePublishProxy.PublishMessage(Object message, String pubsubName, String topicName) in /src/Shared/MyNamespace/Proxies/MessagePublish/MessagePublishProxy.cs:line 109
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.DelayPromise.CompleteTimedOut()
2024-10-11T10:58:19-04:00          at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
2024-10-11T10:58:19-04:00          at System.Threading.TimerQueue.FireNextTimers()
2024-10-11T10:58:19-04:00          at System.Threading.ThreadPoolWorkQueue.Dispatch()
2024-10-11T10:58:19-04:00          at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
2024-10-11T10:58:19-04:00       --- End of stack trace from previous location ---
2024-10-11T10:58:19-04:00          at Dapr.Client.DaprClientGrpc.MakePublishRequest(String pubsubName, String topicName, ByteString content, Dictionary`2 metadata, String dataContentType, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at MyNamespace.Proxies.StatusChange.MessagePublishProxy.PublishMessage(Object message, String pubsubName, String topicName) in /src/Shared/MyNamespace/Proxies/MessagePublish/MessagePublishProxy.cs:line 109
2024-10-11T10:58:19-04:00 warn: MyNamespace.Proxies.StatusChange.MessagePublishProxy[0]
2024-10-11T10:58:19-04:00       Disposing DaprClient.
2024-10-11T10:58:19-04:00 warn: MyNamespace.Proxies.StatusChange.MessagePublishProxy[0]
2024-10-11T10:58:19-04:00       Building new DaprClient.
2024-10-11T10:58:19-04:00 fail: MyNamespace.Proxies.StatusChange.MessagePublishProxy[0]
2024-10-11T10:58:19-04:00       Exception in PublishMessage.
2024-10-11T10:58:19-04:00       System.OperationCanceledException: The operation was canceled.
2024-10-11T10:58:19-04:00        ---> System.OperationCanceledException: The operation was canceled.
2024-10-11T10:58:19-04:00          at System.Threading.CancellationToken.ThrowOperationCanceledException()
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
2024-10-11T10:58:19-04:00          --- End of inner exception stack trace ---
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.ResolveException(String summary, Exception ex, Nullable`1& status, Exception& resolvedException)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.RunCall(HttpRequestMessage request, Nullable`1 timeout)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.GrpcCall`2.StartUnaryCore(HttpContent content)
2024-10-11T10:58:19-04:00          at Grpc.Net.Client.Internal.HttpClientCallInvoker.AsyncUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
2024-10-11T10:58:19-04:00          at Grpc.Core.Interceptors.InterceptingCallInvoker.<AsyncUnaryCall>b__4_0[TRequest,TResponse](TRequest req, ClientInterceptorContext`2 ctx)
2024-10-11T10:58:19-04:00          at Grpc.Core.Interceptors.InterceptingCallInvoker.AsyncUnaryCall[TRequest,TResponse](Method`2 method, String host, CallOptions options, TRequest request)
2024-10-11T10:58:19-04:00          at Dapr.Client.DaprClientGrpc.MakePublishRequest(String pubsubName, String topicName, ByteString content, Dictionary`2 metadata, String dataContentType, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start[TStateMachine](TStateMachine& stateMachine)
2024-10-11T10:58:19-04:00          at Dapr.Client.DaprClientGrpc.PublishEventAsync[TData](String pubsubName, String topicName, TData data, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at MyNamespace.Proxies.StatusChange.MessagePublishProxy.PublishMessage(Object message, String pubsubName, String topicName) in /src/Shared/MyNamespace/Proxies/MessagePublish/MessagePublishProxy.cs:line 109
2024-10-11T10:58:19-04:00          at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
2024-10-11T10:58:19-04:00          at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.AwaitTaskContinuation.RunOrScheduleAction(IAsyncStateMachineBox box, Boolean allowInlining)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.RunContinuations(Object continuationObject)
2024-10-11T10:58:19-04:00          at System.Threading.Tasks.Task.DelayPromise.CompleteTimedOut()
2024-10-11T10:58:19-04:00          at System.Threading.TimerQueueTimer.Fire(Boolean isThreadPool)
2024-10-11T10:58:19-04:00          at System.Threading.TimerQueue.FireNextTimers()
2024-10-11T10:58:19-04:00          at System.Threading.ThreadPoolWorkQueue.Dispatch()
2024-10-11T10:58:19-04:00          at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
2024-10-11T10:58:19-04:00       --- End of stack trace from previous location ---
2024-10-11T10:58:19-04:00          at Dapr.Client.DaprClientGrpc.MakePublishRequest(String pubsubName, String topicName, ByteString content, Dictionary`2 metadata, String dataContentType, CancellationToken cancellationToken)
2024-10-11T10:58:19-04:00          at MyNamespace.Proxies.StatusChange.MessagePublishProxy.PublishMessage(Object message, String pubsubName, String topicName) in /src/Shared/MyNamespace/Proxies/MessagePublish/MessagePublishProxy.cs:line 109
2024-10-11T10:58:19-04:00 fail: MyNamespace.Proxies.StatusChange.MessagePublishProxy[0]
2024-10-11T10:58:19-04:00       PublishEventAsync failed 3 times. This message will be lost forever.
github-actions[bot] commented 1 week ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

MattCosturos commented 1 week ago

Not stale