Rebus stops processing messages

[sigurdhj]

Hi.

I'm having an issue where Rebus stops processing messages. It seems to happen after the app has been running for a while. If I restart my app the messages get processed, but after a while the issue reappears. The log messages I've gotten are:

warn: Rebus.Workers.ThreadPoolBased.ThreadPoolWorker[0]
      An error occurred when attempting to receive the next message: Experimental.System.Messaging.MessageQueueException (0x80004005)
         at Experimental.System.Messaging.MessageQueueTransaction.Begin()
         at Rebus.Msmq.Experimental.MsmqTransport.Receive(ITransactionContext context, CancellationToken cancellationToken)
         at Rebus.Workers.ThreadPoolBased.ThreadPoolWorker.ReceiveTransportMessage(CancellationToken token, ITransactionContext context)
fail: Rebus.Workers.ThreadPoolBased.ThreadPoolWorker[0]
      Unhandled exception in worker "Rebus 5 worker 1"
      System.AggregateException: An error occurred while writing to logger(s). (The RPC server is unavailable.)
       ---> System.ComponentModel.Win32Exception (1722): The RPC server is unavailable.
         at System.Diagnostics.EventLogInternal.InternalWriteEvent(UInt32 eventID, UInt16 category, EventLogEntryType type, String[] strings, Byte[] rawData, String currentMachineName)
         at System.Diagnostics.EventLogInternal.WriteEvent(EventInstance instance, Byte[] data, Object[] values)
         at Microsoft.Extensions.Logging.EventLog.WindowsEventLog.WriteEntry(String message, EventLogEntryType type, Int32 eventID, Int16 category)
         at Microsoft.Extensions.Logging.EventLog.EventLogLogger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
         at Microsoft.Extensions.Logging.Logger.<Log>g__LoggerLog|14_0[TState](LogLevel logLevel, EventId eventId, ILogger logger, Exception exception, Func`3 formatter, List`1& exceptions, TState& state)
         --- End of inner exception stack trace ---
         at Microsoft.Extensions.Logging.Logger.ThrowLoggingError(List`1 exceptions)
         at Microsoft.Extensions.Logging.Logger.Log[TState](LogLevel logLevel, EventId eventId, TState state, Exception exception, Func`3 formatter)
         at Microsoft.Extensions.Logging.LoggerExtensions.Log(ILogger logger, LogLevel logLevel, EventId eventId, Exception exception, String message, Object[] args)
         at Rebus.ServiceProvider.Internals.MicrosoftExtensionsLoggingLoggerFactory.MicrosoftExtensionsLoggingLogger.Warn(String message, Object[] objs)
         at Rebus.Workers.ThreadPoolBased.ThreadPoolWorker.ReceiveTransportMessage(CancellationToken token, ITransactionContext context)
         at Rebus.Workers.ThreadPoolBased.ThreadPoolWorker.TryAsyncReceive(CancellationToken token, IDisposable parallelOperation)

My best guess is that when "The RPC server is unavailable"-issue happens, Rebus cannot recover and just sits there while the queue messages builds up.

My app is running on .NET 8 on Windows. MSMQ is installed on the same machine as the app is running.

Can you help me troubleshoot this further?

[mookid8000]

Weird!

It looks like MessageQueueException (0x80004005) means that the process does not have the required privileges to access MSMQ: https://stackoverflow.com/questions/6511865/messagequeueexception-0x80004005-access-to-message-queuing-system-is-denied

Also the error log that follows: "System.AggregateException: An error occurred while writing to logger(s). (The RPC server is unavailable.)" seems to be an error that you can encounter when the process does not have the required privileges to write to the Windows Event Log.

I am just guessing wildly here, but are you by any chance using Kerberos authentication? and could your machine have a problem accessing the local Active Directory? Because I believe this could actually make the process' privileges expire after a while (e.g. like 10 hours), which could probably give you errors like those you are seeing. I am in doubt though whether this could only be a problem if you're using "public queues" (i.e. MSMQ queues visible in Active Directory, as opposed to "private queues" which are your normal MSMQ queues)

If none of this sounds like it could be the problem, maybe you could switch to the other experimental MSMQ transport? I've been using Rebus.Msmq.Messaging without any issues for most of 2024.

[sigurdhj]

I switched to Rebus.Msmq.Messaging yesterday and the issue seems to be gone. Will keep testing for a few days to verify.

[sigurdhj]

Solved by switching to Rebus.Msmq.Messaging.

[mookid8000]

Excellent! Thanks for reporting back here with your findings.