Getting "Pipelining of requests forbidden" and "The operation has timed out." under load

[TheBlaze]

Hi,

We started using Rebus and we are getting two errors in our performance testing: "Pipelining of requests forbidden" and "The operation has timed out." We are using the latest version of Rebus (6.6.1) and Rebus.RabbitMq (7.3.0).

The setup: _bus = Configure.With(_activator) .Transport(t => t.UseRabbitMqAsOneWayClient(new List() { endpoint })) .Logging(l => l.Log4Net()) .Serialization(s => s.Register(c => new Serializing.MessageSerializer(c.Get()))) .Start();

The call (the same bus instance is used for all calls): _bus.Advanced.Routing.Send(destinationQueue, message, headers).GetAwaiter().GetResult();

The errors: Error=System.NotSupportedException: Pipelining of requests forbidden at RabbitMQ.Client.Impl.RpcContinuationQueue.Enqueue(IRpcContinuation k) at RabbitMQ.Client.Impl.ModelBase.Enqueue(IRpcContinuation k) at RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body) at RabbitMQ.Client.Framing.Impl.Model._Private_ConfirmSelect(Boolean nowait) at Rebus.RabbitMq.RabbitMqTransport.DoSend(ConcurrentQueue'1 outgoingMessages, IModel model, Boolean isExpress) at Rebus.RabbitMq.RabbitMqTransport.d54.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Rebus.Transport.TransactionContext.d32.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Rebus.Transport.TransactionContext.d27.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Rebus.Bus.RebusBus.d43.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at IGP.TechnicalServices.QueueManagement.Publishing.QueuePublisher.Publish(Object message, String destinationQueue, List'1 retryProfile, Dictionary'2 headers)

and: Error=System.TimeoutException: The operation has timed out. at RabbitMQ.Util.BlockingCell'1.WaitForValue(TimeSpan timeout) at RabbitMQ.Client.Impl.SimpleBlockingRpcContinuation.GetReply(TimeSpan timeout) at RabbitMQ.Client.Impl.ModelBase.ModelRpc(MethodBase method, ContentHeaderBase header, Byte[] body) at RabbitMQ.Client.Framing.Impl.Model._Private_ConfirmSelect(Boolean nowait) at Rebus.RabbitMq.RabbitMqTransport.DoSend(ConcurrentQueue'1 outgoingMessages, IModel model, Boolean isExpress) at Rebus.RabbitMq.RabbitMqTransport.d54.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Rebus.Transport.TransactionContext.d32.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Rebus.Transport.TransactionContext.d27.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Rebus.Bus.RebusBus.d43.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at IGP.TechnicalServices.QueueManagement.Publishing.QueuePublisher.Publish(Object message, String destinationQueue, List'1 retryProfile, Dictionary'2 headers)

Are we doing something wrong? Is there a configuration we should be setting? Longer timeout?

[mookid8000]

Could you try and update the RabbitMQ transport to the latest prerelease version?

update-package rebus.rabbitmq -pre

?

IIRC something was done in one of the prerelease versions that fixes this particular issue.

[TheBlaze]

Hello,

Using 7.3.2-b01 solves "Pipelining of requests forbidden" errors, but timeouts still happen, performance is worse and authentication fails if password contains special chars.

I think timeouts are connected to this: https://github.com/rabbitmq/rabbitmq-dotnet-client/issues/959

Do you have any suggestion for authentication? How should special chars ("-", "@",...) be handled? HttpUtility.UrlEncode() doesn't work (any more).

[TheBlaze]

Solved authentication problems using "CustomizeConnectionFactory" and setting username and password there.

[TheBlaze]

Unfortunately 7.3.2-b01 is too slow and does not solve the timeout errors so we won't be using this version. What ultimately solved the issue was setting minimal number of threads in thread pool to a higher value (ThreadPool.SetMinThreads(5000, minIOC)).