search

dotnet / orleans

Cloud Native application framework for .NET
https://docs.microsoft.com/dotnet/orleans
MIT License
9.94k stars 2.02k forks source link

Orleans Silos freezing and crashing #9036

Open runenilsenoe opened 1 month ago

runenilsenoe commented 1 month ago

Context: Since we’ve migrated to the Orleans 7, we’ve experienced a few performance issues..

We are running orleans in kubernetes with kubernets hosting and clustering. But after the migration we have had issues with silos not responding. Not much exceptions just timing out. We have added callthreadreentrancy where needed but it still is not reliable. Healthprobing them and killing is helping but the system is down until the failing probe threshold is hit, so its not a good fix. We probably have some issues in our code but its hard to say where to start looking when the silos just die.

2024-06-04 12:41:10.476 | {"Message":"Exception publishing client routing table to silo \"S10.244.6.9:11111:76427299\"","MessageTemplate":"Exception publishing client routing table to silo {SiloAddress}","Exception":{"Type":"Orleans.Runtime.OrleansMessageRejectionException","Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.9:11111:76427299, will retry after 895.3856ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","StackTrace":"   at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 98\n   at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.ClientDirectory.PublishUpdates() in /_/src/Orleans.Runtime/GrainDirectory/ClientDirectory.cs:line 499"},"SiloAddress":"S10.244.6.9:11111:76427299","ExceptionDetail":{"HResult":-2146233088,"Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.9:11111:76427299, will retry after 895.3856ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","Source":"System.Private.CoreLib","TargetSite":"Void Throw()","Type":"Orleans.Runtime.OrleansMessageRejectionException"},"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-58646859d8-5htvm"} |  
  |   | 2024-06-04 12:41:10.385 | {"Message":"Indirect probe request #60 to silo \"S10.244.6.9:11111:76427299\" via silo \"S10.244.5.16:11111:76502162\" failed after 00:00:01.5819712 with a direct probe response time of 00:00:01.5604363. Failure message: \"Encountered exception \nExc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.6.9:11111:76427299. See InnerException\n ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.6.9:11111. Error: HostUnreachable\n   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 65\n   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 61\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 193\n   --- End of inner exception stack trace ---\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 221\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 106\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226\n   at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 98\n   at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Internal.OrleansTaskExtentions.WithTimeout(Task taskToComplete, TimeSpan timeout, String exceptionMessage) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 87\n   at Orleans.Runtime.MembershipService.MembershipSystemTarget.ProbeIndirectly(SiloAddress target, TimeSpan probeTimeout, Int32 probeNumber) in /_/src/Orleans.Runtime/MembershipService/MembershipSystemTarget.cs:line 86\". Intermediary health score: 0","MessageTemplate":"Indirect probe request #{Id} to silo {SiloAddress} via silo {IntermediarySiloAddress} failed after {RoundTripTime} with a direct probe response time of {ProbeResponseTime}. Failure message: {FailureMessage}. Intermediary health score: {IntermediaryHealthScore}","Id":60,"SiloAddress":"S10.244.6.9:11111:76427299","IntermediarySiloAddress":"S10.244.5.16:11111:76502162","RoundTripTime":"00:00:01.5819712","ProbeResponseTime":"00:00:01.5604363","FailureMessage":"Encountered exception \nExc level 0: Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.6.9:11111:76427299. See InnerException\n ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.6.9:11111. Error: HostUnreachable\n   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 65\n   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 61\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 193\n   --- End of inner exception stack trace ---\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 221\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 106\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226\n   at Orleans.Serialization.Invocation.ResponseCompletionSource.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 98\n   at System.Threading.Tasks.ValueTask.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Internal.OrleansTaskExtentions.WithTimeout(Task taskToComplete, TimeSpan timeout, String exceptionMessage) in /_/src/Orleans.Core/Async/TaskExtensions.cs:line 87\n   at Orleans.Runtime.MembershipService.MembershipSystemTarget.ProbeIndirectly(SiloAddress target, TimeSpan probeTimeout, Int32 probeNumber) in /_/src/Orleans.Runtime/MembershipService/MembershipSystemTarget.cs:line 86","IntermediaryHealthScore":0,"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-58646859d8-96gfk"} |  
  |   | 2024-06-04 12:41:10.372 | {"Message":"Connection id \"\"0HN44G8KKFI9R\"\", Request id \"\"0HN44G8KKFI9R:000000E9\"\": An unhandled exception was thrown by the application.","MessageTemplate":"Connection id \"{ConnectionId}\", Request id \"{TraceIdentifier}\": An unhandled exception was thrown by the application.","Exception":{"Type":"Orleans.Runtime.OrleansMessageRejectionException","Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.6.9:11111:76427299. See InnerException\n ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.6.9:11111. Error: HostUnreachable\n   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 65\n   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 61\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 193\n   --- End of inner exception stack trace ---\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 221\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 106\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","StackTrace":"   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at OrleansDashboard.DashboardClient.ClusterStats()\n   at OrleansDashboard.DashboardMiddleware.Invoke(HttpContext context)\n   at Microsoft.AspNetCore.Builder.Extensions.MapMiddleware.InvokeCore(HttpContext context, PathString matchedPath, PathString remainingPath)\n   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)"},"ConnectionId":"0HN44G8KKFI9R","TraceIdentifier":"0HN44G8KKFI9R:000000E9","EventId":{"Id":13,"Name":"ApplicationError"},"RequestId":"0HN44G8KKFI9R:000000E9","RequestPath":"/dashboard/ClusterStats","ExceptionDetail":{"HResult":-2146233088,"Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to endpoint S10.244.6.9:11111:76427299. See InnerException\n ---> Orleans.Networking.Shared.SocketConnectionException: Unable to connect to 10.244.6.9:11111. Error: HostUnreachable\n   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 65\n   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 61\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 193\n   --- End of inner exception stack trace ---\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 221\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 106\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","Source":"System.Private.CoreLib","TargetSite":"Void Throw()","Type":"Orleans.Runtime.OrleansMessageRejectionException"},"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-58646859d8-96gfk"} |  
  |   | 2024-06-04 12:41:10.371 | {"Message":"Connection attempt to endpoint \"S10.244.6.9:11111:76427299\" failed","MessageTemplate":"Connection attempt to endpoint {EndPoint} failed","Exception":{"Type":"Orleans.Networking.Shared.SocketConnectionException","Message":"Unable to connect to 10.244.6.9:11111. Error: HostUnreachable","StackTrace":"   at Orleans.Networking.Shared.SocketConnectionFactory.ConnectAsync(EndPoint endpoint, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/Shared/SocketConnectionFactory.cs:line 65\n   at Orleans.Runtime.Messaging.ConnectionFactory.ConnectAsync(SiloAddress address, CancellationToken cancellationToken) in /_/src/Orleans.Core/Networking/ConnectionFactory.cs:line 61\n   at Orleans.Runtime.Messaging.ConnectionManager.ConnectAsync(SiloAddress address, ConnectionEntry entry) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 193"},"EndPoint":"S10.244.6.9:11111:76427299","ExceptionDetail":{"HResult":-2146233088,"Message":"Unable to connect to 10.244.6.9:11111. Error: HostUnreachable","Source":"Orleans.Core","TargetSite":"Void MoveNext()","Type":"Orleans.Networking.Shared.SocketConnectionException"},"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-58646859d8-5htvm"} |  
 

I dont have many proposals for a next step here, as we have tried to scour the solution for reentrancy issues.

runenilsenoe commented 1 month ago 

2024-06-05 13:58:58.922 | {"Message":"Created \"ActivationTaskScheduler-3:Queued=0\" with GrainContext=\"[SystemTarget: S10.244.6.40:11111:76593537/sys.svc.user.36F5F3BF/10.244.6.40:11111@76593537@b81578bf00000000877f5b7300000000]\"","MessageTemplate":"Created {TaskScheduler} with GrainContext={GrainContext}","TaskScheduler":"ActivationTaskScheduler-3:Queued=0","GrainContext":"[SystemTarget: S10.244.6.40:11111:76593537/sys.svc.user.36F5F3BF/10.244.6.40:11111@76593537@b81578bf00000000877f5b7300000000]","app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-mkd8v"} |  
  |   | 2024-06-05 13:58:58.916 | {"Message":"Monitoring cluster membership updates","MessageTemplate":"Monitoring cluster membership updates","app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-mkd8v"} |  
  |   | 2024-06-05 13:58:58.911 | {"Message":"Created \"ActivationTaskScheduler-2:Queued=0\" with GrainContext=\"[SystemTarget: S10.244.6.40:11111:76593537/sys.svc.migrator/10.244.6.40:11111@76593537@e8f4d5ad00000000877f5b7300000000]\"","MessageTemplate":"Created {TaskScheduler} with GrainContext={GrainContext}","TaskScheduler":"ActivationTaskScheduler-2:Queued=0","GrainContext":"[SystemTarget: S10.244.6.40:11111:76593537/sys.svc.migrator/10.244.6.40:11111@76593537@e8f4d5ad00000000877f5b7300000000]","app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-mkd8v"} |  
  |   | 2024-06-05 13:58:58.824 | {"Message":"Starting \"VirtualBucketsRingProvider\" on silo \"S10.244.6.40:11111:76593537/x9F156319\".","MessageTemplate":"Starting {Name} on silo {SiloAddress}.","Name":"VirtualBucketsRingProvider","SiloAddress":"S10.244.6.40:11111:76593537/x9F156319","app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-mkd8v"} |  
  |   | 2024-06-05 13:58:58.808 | {"Message":"Created \"ActivationTaskScheduler-1:Queued=0\" with GrainContext=\"[SystemTarget: S10.244.6.40:11111:76593537/sys.svc.catalog/10.244.6.40:11111@76593537@035e1f7900000000877f5b7300000000]\"","MessageTemplate":"Created {TaskScheduler} with GrainContext={GrainContext}","TaskScheduler":"ActivationTaskScheduler-1:Queued=0","GrainContext":"[SystemTarget: S10.244.6.40:11111:76593537/sys.svc.catalog/10.244.6.40:11111@76593537@035e1f7900000000877f5b7300000000]","app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-mkd8v"} |  
  |   | 2024-06-05 13:58:58.765 | {"Message":"Exception during Grain method call of message \"ReadOnly IsAlwaysInterleave Request [S10.244.6.41:11111:76586547 sys.client/313d93c3d4af4ddeb4e59342800160c2]->[S10.244.8.130:11111:76586541 grantproject/4a02897701b94d04acb30a37a0a4dccd] Innbyggertjenester.GrainInterfaces.Donatio.IGrantProjectGrain[(Innbyggertjenester.GrainInterfaces.Donatio.IGrantProjectGrain)Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain].GetApplicationView() #6904\": ","MessageTemplate":"Exception during Grain method call of message {Message}: ","Exception":{"Type":"Orleans.Runtime.OrleansMessageRejectionException","Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","StackTrace":"   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 732\n   at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29\n   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 364\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471\n   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplication(Grant grant) in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 59\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplicationView() in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 296\n   at Orleans.Runtime.TaskRequest`1.CompleteInvokeAsync(Task`1 resultTask)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 132\n   at OrleansDashboard.Implementation.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Innbyggertjenester.Silo.ExceptionConversionFilter.Invoke(IIncomingGrainCallContext context) in /app/src/Innbyggertjenester.Silo/ExceptionConversionFilter.cs:line 51\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Orleans.Runtime.InsideRuntimeClient.Invoke(IGrainContext target, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 263"},"_Message":"ReadOnly IsAlwaysInterleave Request [S10.244.6.41:11111:76586547 sys.client/313d93c3d4af4ddeb4e59342800160c2]->[S10.244.8.130:11111:76586541 grantproject/4a02897701b94d04acb30a37a0a4dccd] Innbyggertjenester.GrainInterfaces.Donatio.IGrantProjectGrain[(Innbyggertjenester.GrainInterfaces.Donatio.IGrantProjectGrain)Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain].GetApplicationView() #6904","EventId":{"Id":100322},"ExceptionDetail":{"HResult":-2146233088,"Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","Source":"System.Private.CoreLib","TargetSite":"Void Throw()","Type":"Orleans.Runtime.OrleansMessageRejectionException"},"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-k9tmk"} |  
  |   | 2024-06-05 13:58:58.764 | {"Message":"HandleMessage \"ReadOnly IsAlwaysInterleave Unrecoverable Rejection (info:  Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226\n   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 732\n   at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29\n   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 364\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471) Response [S10.244.8.130:11111:76586541 grantapplication/5d9cbe2600254c048a31e683585864f9]->[S10.244.8.130:11111:76586541 grantproject/4a02897701b94d04acb30a37a0a4dccd] Orleans.Runtime.RejectionResponse #37288\"","MessageTemplate":"HandleMessage {Message}","_Message":"ReadOnly IsAlwaysInterleave Unrecoverable Rejection (info:  Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226\n   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 732\n   at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29\n   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 364\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471) Response [S10.244.8.130:11111:76586541 grantapplication/5d9cbe2600254c048a31e683585864f9]->[S10.244.8.130:11111:76586541 grantproject/4a02897701b94d04acb30a37a0a4dccd] Orleans.Runtime.RejectionResponse #37288","EventId":{"Id":101523},"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-k9tmk"} |  
  |   | 2024-06-05 13:58:58.764 | {"Message":"Creating Unrecoverable rejection with info '\" Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226\n   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 732\n   at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29\n   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 364\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471\"' at:\n\"   at Orleans.Runtime.MessageFactory.CreateRejectionResponse(Message request, RejectionTypes type, String info, Exception ex)\n   at Orleans.Runtime.Messaging.MessageCenter.RejectMessage(Message message, RejectionTypes rejectionType, Exception exc, String rejectInfo)\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__OnAddressingFailure\|39_1(Message m, Exception ex)\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m)\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)\n   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)\n   at System.Threading.Tasks.TaskSchedulerAwaitTaskContinuation.<>c.<Run>b__2_0(Object state)\n   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)\n   at Orleans.Runtime.Scheduler.WorkItemGroup.Execute()\n   at System.Threading.ThreadPoolWorkQueue.Dispatch()\n   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()\n\"","MessageTemplate":"Creating {RejectionType} rejection with info '{Info}' at:\n{StackTrace}","Exception":{"Type":"Orleans.Runtime.OrleansMessageRejectionException","Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","StackTrace":"   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 732\n   at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29\n   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 364\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471\n   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplication(Grant grant) in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 59\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplicationView() in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 296\n   at Orleans.Runtime.TaskRequest`1.CompleteInvokeAsync(Task`1 resultTask)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 132\n   at OrleansDashboard.Implementation.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Innbyggertjenester.Silo.ExceptionConversionFilter.Invoke(IIncomingGrainCallContext context) in /app/src/Innbyggertjenester.Silo/ExceptionConversionFilter.cs:line 51\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Orleans.Runtime.InsideRuntimeClient.Invoke(IGrainContext target, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 263"},"RejectionType":"Unrecoverable","Info":" Orleans.Runtime.OrleansMessageRejectionException: Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226\n   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 732\n   at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29\n   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 364\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471","StackTrace":"   at Orleans.Runtime.MessageFactory.CreateRejectionResponse(Message request, RejectionTypes type, String info, Exception ex)\n   at Orleans.Runtime.Messaging.MessageCenter.RejectMessage(Message message, RejectionTypes rejectionType, Exception exc, String rejectInfo)\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__OnAddressingFailure\|39_1(Message m, Exception ex)\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m)\n   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)\n   at System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1.AsyncStateMachineBox`1.MoveNext(Thread threadPoolThread)\n   at System.Threading.Tasks.TaskSchedulerAwaitTaskContinuation.<>c.<Run>b__2_0(Object state)\n   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(Task& currentTaskSlot, Thread threadPoolThread)\n   at Orleans.Runtime.Scheduler.WorkItemGroup.Execute()\n   at System.Threading.ThreadPoolWorkQueue.Dispatch()\n   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()\n","ExceptionDetail":{"HResult":-2146233088,"Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","Source":"System.Private.CoreLib","TargetSite":"Void Throw()","Type":"Orleans.Runtime.OrleansMessageRejectionException"},"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-k9tmk"} |  
  |   | 2024-06-05 13:58:58.764 | {"Message":"Rejected message \"ReadOnly IsAlwaysInterleave Request [S10.244.8.130:11111:76586541 grantproject/4a02897701b94d04acb30a37a0a4dccd]->[ grantapplication/5d9cbe2600254c048a31e683585864f9] Innbyggertjenester.GrainInterfaces.Donatio.IGrantApplicationGrain.Get(Innbyggertjenester.GrainInterfaces.Donatio.Models.Grant.Grant) #37288\" with reason 'null' (Unrecoverable)","MessageTemplate":"Rejected message {Message} with reason '{Reason}' ({RejectionType})","Exception":{"Type":"Orleans.Runtime.OrleansMessageRejectionException","Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","StackTrace":"   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 732\n   at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29\n   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 364\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471\n   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplication(Grant grant) in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 59\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplicationView() in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 296\n   at Orleans.Runtime.TaskRequest`1.CompleteInvokeAsync(Task`1 resultTask)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 132\n   at OrleansDashboard.Implementation.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Innbyggertjenester.Silo.ExceptionConversionFilter.Invoke(IIncomingGrainCallContext context) in /app/src/Innbyggertjenester.Silo/ExceptionConversionFilter.cs:line 51\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Orleans.Runtime.InsideRuntimeClient.Invoke(IGrainContext target, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 263"},"_Message":"ReadOnly IsAlwaysInterleave Request [S10.244.8.130:11111:76586541 grantproject/4a02897701b94d04acb30a37a0a4dccd]->[ grantapplication/5d9cbe2600254c048a31e683585864f9] Innbyggertjenester.GrainInterfaces.Donatio.IGrantApplicationGrain.Get(Innbyggertjenester.GrainInterfaces.Donatio.Models.Grant.Grant) #37288","Reason":null,"RejectionType":"Unrecoverable","EventId":{"Id":101042,"Name":"Orleans.Messaging.Dispatcher.Rejected"},"ExceptionDetail":{"HResult":-2146233088,"Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","Source":"System.Private.CoreLib","TargetSite":"Void Throw()","Type":"Orleans.Runtime.OrleansMessageRejectionException"},"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-k9tmk"} |  
  |   | 2024-06-05 13:58:58.764 | {"Message":"Failed to address message \"ReadOnly IsAlwaysInterleave Request [S10.244.8.130:11111:76586541 grantproject/4a02897701b94d04acb30a37a0a4dccd]->[ grantapplication/5d9cbe2600254c048a31e683585864f9] Innbyggertjenester.GrainInterfaces.Donatio.IGrantApplicationGrain.Get(Innbyggertjenester.GrainInterfaces.Donatio.Models.Grant.Grant) #37288\"","MessageTemplate":"Failed to address message {Message}","Exception":{"Type":"Orleans.Runtime.OrleansMessageRejectionException","Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","StackTrace":"   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 732\n   at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29\n   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 364\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471\n   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplication(Grant grant) in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 59\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplicationView() in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 296\n   at Orleans.Runtime.TaskRequest`1.CompleteInvokeAsync(Task`1 resultTask)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 132\n   at OrleansDashboard.Implementation.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Innbyggertjenester.Silo.ExceptionConversionFilter.Invoke(IIncomingGrainCallContext context) in /app/src/Innbyggertjenester.Silo/ExceptionConversionFilter.cs:line 51\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Orleans.Runtime.InsideRuntimeClient.Invoke(IGrainContext target, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 263"},"_Message":"ReadOnly IsAlwaysInterleave Request [S10.244.8.130:11111:76586541 grantproject/4a02897701b94d04acb30a37a0a4dccd]->[ grantapplication/5d9cbe2600254c048a31e683585864f9] Innbyggertjenester.GrainInterfaces.Donatio.IGrantApplicationGrain.Get(Innbyggertjenester.GrainInterfaces.Donatio.Models.Grant.Grant) #37288","EventId":{"Id":100071,"Name":"Orleans.Messaging.Dispatcher.SelectTargetFailed"},"ExceptionDetail":{"HResult":-2146233088,"Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.7159ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","Source":"System.Private.CoreLib","TargetSite":"Void Throw()","Type":"Orleans.Runtime.OrleansMessageRejectionException"},"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-k9tmk"} |  
  |   | 2024-06-05 13:58:58.763 | {"Message":"Exception during Grain method call of message \"ReadOnly IsAlwaysInterleave Request [S10.244.6.41:11111:76586547 sys.client/313d93c3d4af4ddeb4e59342800160c2]->[S10.244.8.130:11111:76586541 grantproject/d83b72eaf9e241548bfa48a8e49951f7] Innbyggertjenester.GrainInterfaces.Donatio.IGrantProjectGrain[(Innbyggertjenester.GrainInterfaces.Donatio.IGrantProjectGrain)Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain].GetApplicationView() #6896\": ","MessageTemplate":"Exception during Grain method call of message {Message}: ","Exception":{"Type":"Orleans.Runtime.OrleansMessageRejectionException","Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.3011ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","StackTrace":"   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Orleans.Runtime.GrainDirectory.LocalGrainDirectory.LookupAsync(GrainId grainId, Int32 hopCount) in /_/src/Orleans.Runtime/GrainDirectory/LocalGrainDirectory.cs:line 732\n   at Orleans.Runtime.GrainDirectory.DhtGrainLocator.Lookup(GrainId grainId) in /_/src/Orleans.Runtime/GrainDirectory/DhtGrainLocator.cs:line 29\n   at Orleans.Runtime.Placement.PlacementService.PlacementWorker.GetOrPlaceActivationAsync(Message firstMessage) in /_/src/Orleans.Runtime/Placement/PlacementService.cs:line 364\n   at Orleans.Runtime.Messaging.MessageCenter.<AddressAndSendMessage>g__SendMessageAsync\|39_0(Task addressMessageTask, Message m) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 471\n   at Orleans.Serialization.Invocation.ResponseCompletionSource`1.GetResult(Int16 token) in /_/src/Orleans.Serialization/Invocation/ResponseCompletionSource.cs:line 230\n   at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)\n--- End of stack trace from previous location ---\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplication(Grant grant) in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 59\n   at Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain.GetApplicationView() in /app/src/Innbyggertjenester.Grains/Donatio/StatefulGrains/GrantProjectGrain.cs:line 296\n   at Orleans.Runtime.TaskRequest`1.CompleteInvokeAsync(Task`1 resultTask)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 132\n   at OrleansDashboard.Implementation.GrainProfilerFilter.Invoke(IIncomingGrainCallContext context)\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Innbyggertjenester.Silo.ExceptionConversionFilter.Invoke(IIncomingGrainCallContext context) in /app/src/Innbyggertjenester.Silo/ExceptionConversionFilter.cs:line 51\n   at Orleans.Runtime.GrainMethodInvoker.Invoke() in /_/src/Orleans.Runtime/Core/GrainMethodInvoker.cs:line 94\n   at Orleans.Runtime.InsideRuntimeClient.Invoke(IGrainContext target, Message message) in /_/src/Orleans.Runtime/Core/InsideRuntimeClient.cs:line 263"},"_Message":"ReadOnly IsAlwaysInterleave Request [S10.244.6.41:11111:76586547 sys.client/313d93c3d4af4ddeb4e59342800160c2]->[S10.244.8.130:11111:76586541 grantproject/d83b72eaf9e241548bfa48a8e49951f7] Innbyggertjenester.GrainInterfaces.Donatio.IGrantProjectGrain[(Innbyggertjenester.GrainInterfaces.Donatio.IGrantProjectGrain)Innbyggertjenester.Grains.Donatio.StatefulGrains.GrantProjectGrain].GetApplicationView() #6896","EventId":{"Id":100322},"ExceptionDetail":{"HResult":-2146233088,"Message":"Exception while sending message: Orleans.Runtime.Messaging.ConnectionFailedException: Unable to connect to S10.244.6.40:11111:76586541, will retry after 916.3011ms\n   at Orleans.Runtime.Messaging.ConnectionManager.GetConnectionAsync(SiloAddress endpoint) in /_/src/Orleans.Core/Networking/ConnectionManager.cs:line 99\n   at Orleans.Runtime.Messaging.MessageCenter.<SendMessage>g__SendAsync\|29_0(MessageCenter messageCenter, ValueTask`1 connectionTask, Message msg) in /_/src/Orleans.Runtime/Messaging/MessageCenter.cs:line 226","Source":"System.Private.CoreLib","TargetSite":"Void Throw()","Type":"Orleans.Runtime.OrleansMessageRejectionException"},"app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-k9tmk"} |  
  |   | 2024-06-05 13:58:58.761 | {"Message":"Validating budget details","MessageTemplate":"Validating budget details","StackTrace":"GrantApplicationGrain.Get","GrainType":"GrantApplicationGrain","Method":"Get","GrainId":"65662b2454d0413d8a21548378595900","app":"Innbyggertjenester.Silo","ENV":"Production","APP_NAME":"innbyggertjenester-silo-e2e","POD_NAMESPACE":"e2e","POD_NAME":"innbyggertjenester-silo-e2e-5c949dd856-k9tmk"} |  
 ```