search

marcoCasamento / Hangfire.Redis.StackExchange

HangFire Redis storage based on original (and now unsupported) Hangfire.Redis but using lovely StackExchange.Redis client
Other
452 stars 108 forks source link

Slow performance when throwing large exceptions #84

Closed DR9885 closed 3 years ago

DR9885 commented 5 years ago

Large Exceptions causes performance to move slow on failure page, especially when there are more than 5 failures.

Maybe failures should be compressed? I'm taking a blind guess, have not gone through the code.

A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = WritableServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "x.x.x.x:27017" }", EndPoint: "x.x.x.x:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond x.x.x.x:27017 at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at System.Net.Sockets.Socket.<>c.b__271_0(IAsyncResult iar) --- End of stack trace from previous location where exception was thrown --- at MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken) --- End of inner exception stack trace --- at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken) at MongoDB.Driver.Core.Servers.ServerMonitor.HeartbeatAsync(CancellationToken cancellationToken)" }, { ServerId: "{ ClusterId : 1, EndPoint : "10.6.1.31:27017" }", EndPoint: "10.6.1.31:27017", State: "Connected", Type: "ReplicaSetSecondary", WireVersionRange: "[0, 6]" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongo000.perf.280cap.net:27017" }", EndPoint: "Unspecified/mongo000.perf.280cap.net:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.6.1.56:27017 at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult) at System.Net.Sockets.Socket.<>c.b__271_0(IAsyncResult iar) --- End of stack trace from previous location where exception was thrown --- at MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken) at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken) --- End of inner exception stack trace --- at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken) at MongoDB.Driver.Core.Servers.ServerMonitor.HeartbeatAsync(CancellationToken cancellationToken)" }] }.

System.TimeoutException: A timeout occured after 30000ms selecting a server using CompositeServerSelector{ Selectors = WritableServerSelector, LatencyLimitingServerSelector{ AllowedLatencyRange = 00:00:00.0150000 } }. Client view of cluster state is { ClusterId : "1", ConnectionMode : "Automatic", Type : "ReplicaSet", State : "Connected", Servers : [{ ServerId: "{ ClusterId : 1, EndPoint : "x.x.x.x:27017" }", EndPoint: "x.x.x.x:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond x.x.x.x27017
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.Socket.<>c.<ConnectAsync>b__271_0(IAsyncResult iar)
--- End of stack trace from previous location where exception was thrown ---
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.ServerMonitor.HeartbeatAsync(CancellationToken cancellationToken)" }, { ServerId: "{ ClusterId : 1, EndPoint : "10.6.1.31:27017" }", EndPoint: "10.6.1.31:27017", State: "Connected", Type: "ReplicaSetSecondary", WireVersionRange: "[0, 6]" }, { ServerId: "{ ClusterId : 1, EndPoint : "Unspecified/mongo000.perf.280cap.net:27017" }", EndPoint: "Unspecified/mongo000.perf.280cap.net:27017", State: "Disconnected", Type: "Unknown", HeartbeatException: "MongoDB.Driver.MongoConnectionException: An exception occurred while opening a connection to the server. ---> System.Net.Internals.SocketExceptionFactory+ExtendedSocketException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.6.1.56:27017
   at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
   at System.Net.Sockets.Socket.<>c.<ConnectAsync>b__271_0(IAsyncResult iar)
--- End of stack trace from previous location where exception was thrown ---
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.ConnectAsync(Socket socket, EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.TcpStreamFactory.CreateStreamAsync(EndPoint endPoint, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.OpenHelperAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.ServerMonitor.HeartbeatAsync(CancellationToken cancellationToken)" }] }.
   at MongoDB.Driver.Core.Clusters.Cluster.ThrowTimeoutException(IServerSelector selector, ClusterDescription description)
   at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedHelper.HandleCompletedTask(Task completedTask)
   at MongoDB.Driver.Core.Clusters.Cluster.WaitForDescriptionChangedAsync(IServerSelector selector, ClusterDescription description, Task descriptionChangedTask, TimeSpan timeout, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Clusters.Cluster.SelectServerAsync(IServerSelector selector, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Bindings.WritableServerBinding.GetWriteChannelSourceAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.InitializeAsync(CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteContext.CreateAsync(IWriteBinding binding, Boolean retryRequested, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteAsync(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperationAsync[TResult](IWriteBinding binding, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.ExecuteWriteOperationAsync[TResult](IClientSessionHandle session, IWriteOperation`1 operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWriteAsync(IClientSessionHandle session, IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl`1.UsingImplicitSessionAsync[TResult](Func`2 funcAsync, CancellationToken cancellationToken)
   at TwoEightyCap.Util.EventSourcing.Repo.Mongo.MongoCrudRepo.SaveAllAsync[T](String bucketId, IEnumerable`1 entities, CancellationToken ct)
   at TwoEightyCap.Util.EventSourcing.Services.EventService.SaveAllAsync(String bucketId, Event[] events, CancellationToken ct)
   at TwoEightyCap.Job.Security.IceApex.Jobs.BuildJob.SaveEvents(String file, Event[] events, CancellationToken ct) in /home/appveyor/projects/twoeightycap-service-security/src/TwoEightyCap.Job.Security.IceApex/Jobs/BuildJob.cs:line 190
   at TwoEightyCap.Job.Security.IceApex.Jobs.BuildJob.BuildFileAsync(String file, Boolean isBackFill, Int32 batchSize, PerformContext performContext) in /home/appveyor/projects/twoeightycap-service-security/src/TwoEightyCap.Job.Security.IceApex/Jobs/BuildJob.cs:line 171
marcoCasamento commented 3 years ago

That's an idea but I have no relaible way of knowing when the storage is saving "an exception" and furthermore, it would require to be "uncompressed" when reading. At the end of the day it would make that storage implementation slower, while it has been designed for speed. I'd exec redis on a faster server.