search

wdcossey / Apache.NMS.ActiveMQ.NetStd

Apache NMS for ActiveMQ Class Library (.Net Messaging Library Implementation): An implementation of the NMS API for ActiveMQ
Apache License 2.0
14 stars 8 forks source link

Possible slow performance with large messages on Linux #6

Open szyb opened 4 years ago

szyb commented 4 years ago

Hi,

I've been using this library in .NET Core 3.1 and I was surprised that the performance on Windows and Linux is not comparable. After I've made some tests it turns out that when I use quite large message (1MB), then dequeuing it in consumer takes around 19 seconds, while in windows it take few milliseconds.

Here is sample code repository (https://github.com/szyb/activemq_test_netstd) to reproduce the issue, however it requires to have configured Linux & Windows OS with ActiveMQ service on both machines. The configuration of ActiveMQ is pretty standard, the only thing I changed from default is enabling authorization.

I have no idea if this is issue of the library or dotnet core itself. Is it possible to verify that?

wdcossey commented 4 years ago

Hi @szyb ,

I will have a look at this later this evening and get back to you.

wdcossey commented 4 years ago

@szyb ,

Seems there's some differences with how the Sockets work under Linux.

The default buffer size in NMS.ActiveMQ is 8192 bytes, this causes issues with large messages, while I can easily just make the buffer larger this doesn't seem like the appropriate approach (just yet).

Ubuntu Test: image

szyb commented 4 years ago

@wdcossey Thank you for your reply. Do you know which part of the code is responsible for that? Maybe it is related to .NET Core implementation for Linux? The same buffer size works well in Windows.

wdcossey commented 4 years ago

@szyb,

The issue seems to be in BinaryReader.ReadBytes(). _stream.Read() takes a lifetime to complete.

Below is a preview build for you if you want to give it a test, I increased the buffer size to 32kb by default (it was 8kb), anything less (I tried 16kb) than this seems to cause issues under Unix.

https://www.nuget.org/packages/Apache.NMS.ActiveMQ.NetStd/1.7.2.4116-preview

I have added Apache.NMS.ActiveMQ.Transport.Tcp.TcpTransportManager so you can make adjustments to the default buffer size if needed.

i.e.

TcpTransportManager.ReceiveBufferSize = (1024 64); //64kb TcpTransportManager.SendBufferSize = (1024 8); //8kb

szyb commented 4 years ago

@wdcossey I've tested this and I can confirm, everything below 32kb is a disaster on Linux and for 4kb ActiveMQ failed with error: "Channel was inactive for too long". Increasing ReadBufferSize speeds up executing test and at 32kb there are no significant differences for 1MB size. For 10MB it is slightly slower than on Windows, but increasing buffer to 64kb make it equal again. I'm glad that I can manage by my own the buffer size, when I know where the issue is.

Thank you for your time and solution!

wdcossey commented 4 years ago

HI @szyb ,

I am happy that you found a solution, although it's not ideal changing the buffer size depending on the size of the message (i.e. 10Mb).

It's worth nothing that you probably shouldn't be sending large messages over ActiveMQ, while you can it will only slow down more when sending to multiple subscribers (i.e on a topic), ideally your messages should be as small as possible to ensure quick delivery times, not overloading the server's network resources.

I will still play around with this and see what else I can find out about the BinaryReader under Unix.

szyb commented 4 years ago

Hi @wdcossey, I like to have few alternatives so I can choose the best one at the end. This is one of them. Unfortunately I need to send lots of data over the network, so I need to find best balance between size of the package and number of packages. That is why I'm doing such tests :)

I'm really interested in your future findings about the BinaryReader under Unix. I'll try do the same, when time permits.

hpk commented 4 years ago

As another data point, this fixed a performance issue we were having as well. In our case, we have a topic with small messages (less than 1kb) but a decently high message rate (~1k messages/sec).

With the default buffer sizes, our .NET Core consumer on EKS/Docker Linux was unable to keep up with the message volume, causing messages to pile up or get discarded at the broker. The exact same code compiled against .NET Framework and running on Windows had no issues with the message volume.

Switching to 1.7.2.4116-preview and setting the send/receive buffer sizes to 16kb/64kb solved the issue.

wdcossey commented 4 years ago

Thanks for your message @hpk.

I will double check the changes I made, move this out of preview and push an update to nuget.org as soon as possible.

hpk commented 4 years ago

Actually, on further investigation, I was able to solve this by adding the following to my connection URI:

?transport.receiveBufferSize=65536&transport.sendBufferSize=16384

No customization in the code needed.

petertsu commented 3 years ago

I am using failover transport. Connection URI is
failover:(tcp://activemq-0.activemq-cluster:61616,tcp://activemq-1.activemq-cluster:61616,tcp://activemq-2.activemq-cluster:61616)?&transport.receiveBufferSize=65536&transport.sendBufferSize=16384

I am trying to set that properties to the connection URI and get exception.

Apache.NMS.NMSException: No such property or field: receivebuffersize on class: FailoverTransport at Apache.NMS.Util.URISupport.SetProperties(Object target, StringDictionary map, String prefix) at Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransportFactory.CreateTransport(StringDictionary parameters) at Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransportFactory.CreateTransport(CompositeData compositData) at Apache.NMS.ActiveMQ.Transport.Failover.FailoverTransportFactory.doConnect(Uri location) at Apache.NMS.ActiveMQ.ConnectionFactory.CreateActiveMQConnection(String userName, String password) at Votiro.Queue.Internals.ActiveMq.ActiveMqSubscriberAdapterBase.CreateConnection()

wdcossey commented 3 years ago

It's my understanding that the transport options need to be set on the transport url.

Something like the following:

failover:(tcp://activemq-0.activemq-cluster:61616,tcp://activemq-1.activemq-cluster:61616,tcp://activemq-2.activemq-cluster:61616?transport.receiveBufferSize=65536&transport.sendBufferSize=16384)

See here for more information

it's also worth noting that your url mat be malformed.

failover:(tcp://activemq-0.activemq-cluster:61616,tcp://activemq-1.activemq-cluster:61616,tcp://activemq-2.activemq-cluster:61616)?&transport.receiveBufferSize=65536&transport.sendBufferSize=16384

see ?&