RabbitMQ server outage forcers serilog to close

[marrkee]

Hi,

Yesterday after my rabbitmq server outage, my applications stopped logging completely. Here is the stacktrace:

at RabbitMQ.Client.Impl.InboundFrame.ReadFrom(NetworkBinaryReader reader) at RabbitMQ.Client.Impl.SocketFrameHandler.ReadFrame() at RabbitMQ.Client.Framing.Impl.Connection.MainLoopIteration() at RabbitMQ.Client.Framing.Impl.Connection.MainLoop() at RabbitMQ.Client.Impl.SessionBase.Transmit(Command cmd) at RabbitMQ.Client.Impl.ModelBase.ModelSend(MethodBase method, ContentHeaderBase header, Byte[] body) at RabbitMQ.Client.Framing.Impl.Model._Private_BasicPublish(String exchange, String routingKey, Boolean mandatory, IBasicProperties basicProperties, Byte[] body) at RabbitMQ.Client.Impl.ModelBase.BasicPublish(String exchange, String routingKey, Boolean mandatory, IBasicProperties basicProperties, Byte[] body) at RabbitMQ.Client.Impl.AutorecoveringModel.BasicPublish(String exchange, String routingKey, Boolean mandatory, IBasicProperties basicProperties, Byte[] body) at Serilog.Sinks.RabbitMQ.RabbitMQSink.EmitBatch(IEnumerable1 events) at Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink.<EmitBatchAsync>d__15.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Serilog.Sinks.PeriodicBatching.PeriodicBatchingSink.<OnTick>d__16.MoveNext()RabbitMQ.Client.Exceptions.BrokerUnreachableException: None of the specified endpoints were reachable ---> System.AggregateException: One or more errors occurred. ---> RabbitMQ.Client.Exceptions.ConnectFailureException: Connection failed ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 10.132.87.118:5672 at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) at System.Net.Sockets.Socket.Connect(EndPoint remoteEP) at System.Net.Sockets.Socket.Connect(IPAddress address, Int32 port) at RabbitMQ.Client.TcpClientAdapter.<ConnectAsync>d__2.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at RabbitMQ.Client.Impl.TaskExtensions.<TimeoutAfter>d__1.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectOrFail(ITcpClient socket, AmqpTcpEndpoint endpoint, Int32 timeout) --- End of inner exception stack trace --- at RabbitMQ.Client.Impl.SocketFrameHandler.ConnectUsingAddressFamily(AmqpTcpEndpoint endpoint, Func2 socketFactory, Int32 timeout, AddressFamily family) at RabbitMQ.Client.Impl.SocketFrameHandler..ctor(AmqpTcpEndpoint endpoint, Func2 socketFactory, Int32 connectionTimeout, Int32 readTimeout, Int32 writeTimeout) at RabbitMQ.Client.ConnectionFactory.CreateFrameHandler(AmqpTcpEndpoint endpoint) at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func2 selector) --- End of inner exception stack trace --- at RabbitMQ.Client.EndpointResolverExtensions.SelectOne[T](IEndpointResolver resolver, Func2 selector) at RabbitMQ.Client.Framing.Impl.AutorecoveringConnection.Init(IEndpointResolver endpoints) at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName) --- End of inner exception stack trace --- at RabbitMQ.Client.ConnectionFactory.CreateConnection(IEndpointResolver endpointResolver, String clientProvidedName) at RabbitMQ.Client.ConnectionFactory.CreateConnection(IList1 hostnames, String clientProvidedName) at RabbitMQ.Client.ConnectionFactory.CreateConnection(IList1 hostnames) at Serilog.Sinks.RabbitMQ.RabbitMQClient.InitializeEndpoint() at Serilog.Sinks.RabbitMQ.RabbitMQClient..ctor(RabbitMQClientConfiguration configuration) at Serilog.Sinks.RabbitMQ.RabbitMQSink..ctor(RabbitMQClientConfiguration configuration, RabbitMQSinkConfiguration rabbitMQSinkConfiguration) at Serilog.LoggerConfigurationRabbitMqExtension.RabbitMQ(LoggerSinkConfiguration loggerConfiguration, Action2 configure)

Steps to reproduce the behavior:

1. Setup serilog with rabbitMQ
2. Close rabbitMq
3. See error

Ignore and keep logging to file

Additional context here are the logs from my program: Topshelf.Logging.SerilogLogWriter.Fatal() - The service threw an unhandled exception System.IO.FileNotFoundException: Could not load file or assembly 'Microsoft.Diagnostics.Tracing.EventSource, Version=1.1.28.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' or one of its dependencies. The system cannot find the file specified. File name: 'Microsoft.Diagnostics.Tracing.EventSource, Version=1.1.28.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a' at RabbitMQ.Client.ESLog.Error(String message, Exception ex) at RabbitMQ.Client.Framing.Impl.Connection.LogCloseError(String error, Exception ex) at RabbitMQ.Client.Framing.Impl.Connection.ClosingLoop() at RabbitMQ.Client.Framing.Impl.Connection.MainLoop() at System.Threading.ThreadHelper.ThreadStart_Context(Object state) at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx) at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state) at System.Threading.ThreadHelper.ThreadStart()

WRN: Assembly binding logging is turned OFF. To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1. Note: There is some performance penalty associated with assembly bind failure logging. To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

[madslyng]

@marrkee Thanks for the bug report.

To better understand the situation, could you please describe your RabbitMQ setup. I assume it's a single instance of RabbitMQ ?

[madslyng]

Does anyone else have an opinion about this.

From my perspective a RabbitMQ setup should be redundant - and not be a single-point-of-failure.

But I gather from this that, if the RabbitMQ instance is unavailable for a period of time, it should be able to pick up logging once it becomes available again ?