search

gottscj / Hangfire.Mongo

Mongo DB support for Hangfire
MIT License
260 stars 84 forks source link

Connection reset by peer #197

Closed pavloneiman closed 4 years ago

pavloneiman commented 5 years ago

Frequently get Connection reset by peer on background job creation.

Libs:

Environment: Docker image based on ASP.NET Core mcr.microsoft.com/dotnet/core/aspnet:2.2.6-stretch-slim (OS: Debian 9)

MongoDB: version 3.6.12

Exception details:

Hangfire.BackgroundJobClientException: Background job creation failed. See inner exception for details. 
---> MongoDB.Driver.MongoConnectionException: An exception occurred while receiving a message from the server. 
---> System.IO.IOException: Unable to read data from the transport connection: Connection reset by peer. 
---> System.Net.Sockets.SocketException: Connection reset by peer

   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   --- End of inner exception stack trace ---
   at System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)
   at MongoDB.Driver.Core.Misc.StreamExtensionMethods.ReadBytes(Stream stream, Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
   --- End of inner exception stack trace ---
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer()
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveBuffer(Int32 responseTo, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Connections.BinaryConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.ConnectionPools.ExclusiveConnectionPool.AcquiredConnection.ReceiveMessage(Int32 responseTo, IMessageEncoderSelector encoderSelector, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.WireProtocol.CommandUsingCommandMessageWireProtocol<T>.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.WireProtocol.CommandWireProtocol<T>.Execute(IConnection connection, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.ExecuteProtocol[TResult](IWireProtocol<T> protocol, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Servers.Server.ServerChannel.Command[TResult](ICoreSession session, ReadPreference readPreference, DatabaseNamespace databaseNamespace, BsonDocument command, IEnumerable<T> commandPayloads, IElementNameValidator commandValidator, BsonDocument additionalOptions, Action<T> postWriteAction, CommandResponseHandling responseHandling, IBsonSerializer<T> resultSerializer, MessageEncoderSettings messageEncoderSettings, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteCommandOperationBase.ExecuteAttempt(RetryableWriteContext context, Int32 attempt, Nullable<T> transactionNumber, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.RetryableWriteOperationExecutor.Execute[TResult](IRetryableWriteOperation<T> operation, RetryableWriteContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase<T>.ExecuteBatch(RetryableWriteContext context, Batch batch, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase<T>.ExecuteBatches(RetryableWriteContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkUnmixedWriteOperationBase<T>.Execute(RetryableWriteContext context, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.ExecuteBatch(RetryableWriteContext context, Batch batch, CancellationToken cancellationToken)
   at MongoDB.Driver.Core.Operations.BulkMixedWriteOperation.Execute(IWriteBinding binding, CancellationToken cancellationToken)
   at MongoDB.Driver.OperationExecutor.ExecuteWriteOperation[TResult](IWriteBinding binding, IWriteOperation<T> operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl<T>.ExecuteWriteOperation[TResult](IClientSessionHandle session, IWriteOperation<T> operation, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl<T>.BulkWrite(IClientSessionHandle session, IEnumerable<T> requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl<T>.UsingImplicitSession[TResult](Func<T1,T2> func, CancellationToken cancellationToken)
   at MongoDB.Driver.MongoCollectionImpl<T>.BulkWrite(IEnumerable<T> requests, BulkWriteOptions options, CancellationToken cancellationToken)
   at Hangfire.Mongo.MongoWriteOnlyTransaction.Commit()
   at Hangfire.Mongo.MongoConnection.CreateExpiredJob(Job job, IDictionary<T1,T2> parameters, DateTime createdAt, TimeSpan expireIn)
   at async Hangfire.Client.CoreBackgroundJobFactory.RetryOnException[T](Int32& attemptsLeft, Func<T1,T2> action)
--- End of stack trace from previous location where exception was thrown ---
   at Hangfire.Client.CoreBackgroundJobFactory.RetryOnException[T](Int32& attemptsLeft, Func<T1,T2> action)
   at Hangfire.Client.CoreBackgroundJobFactory.Create(CreateContext context)
   at Hangfire.Client.BackgroundJobFactory.<>c__DisplayClass12_0.<CreateWithFilters>b__0()
   at Hangfire.Client.BackgroundJobFactory.InvokeClientFilter(IClientFilter filter, CreatingContext preContext, Func<T> continuation)
   at Hangfire.Client.BackgroundJobFactory.Create(CreateContext context)
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
   --- End of inner exception stack trace ---
   at Hangfire.BackgroundJobClient.Create(Job job, IState state)
   at Hangfire.BackgroundJobClientExtensions.Schedule[T](IBackgroundJobClient client, Expression<T> methodCall, TimeSpan delay)
   at ......
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeNextActionFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Rethrow(ActionExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextExceptionFilterAsync()
gottscj commented 5 years ago

@pavloneiman,

It looks like the underlying socket is being closed? You could try to enable 'trace' logging and log whats going on?