search

Azure / azure-amqp

AMQP C# library
Other
94 stars 72 forks source link

lock contention is extremely high when the request rate is high #129

Open mikelchai opened 5 years ago

mikelchai commented 5 years ago

Timer is used by many functions in the library. What we found is if the request rate is very high, around 30% of CPU is for lock contention. Is there a way to optimize? I know the issue with Timer is fixed in .net core and HashedWheelTimer is implemented in dotnetty.

miso-ms commented 2 years ago

Are there any updates to this issue? We've observed in some of our scenarios similar lock contention by the Amqp producer when targeting .Net Framework 4.7.2.

image

Event Hubs SDK Version: 5.4.1 Target Platform: .Net Framework 4.7.2 Target Event Hubs: >500 partitions, high event rate, low latency

miso-ms commented 2 years ago

Update, learned about a .Net Framework patch that fixes the problematic timer call stack and had some small improvements.

We are still observing (and investigating) lock contention in the amqp WriteBuffer call in this scenario, where a single connection is writing to many partitions (>500): image

xinchen10 commented 2 years ago

The I/O operations on a connection have to be synchronized. There may be some improvements we can do for it, but can you create more connections and distribute the partitions among them? I assume you have 500 senders to the partitions but they share the same connection. If most of them are busy, eventually the connection will be the bottleneck, even if we improve the lock contention for the I/O operations.