search

lcm-proj / lcm

Lightweight Communications and Marshalling
GNU Lesser General Public License v2.1
944 stars 385 forks source link

Losing messages at 30K msg/sec and 50 bytes messages in C++ on linux #365

Closed skdjfnd closed 2 years ago

skdjfnd commented 2 years ago

Hi,

I am using LCM to stream messages between two processes on my local linux machine. My code is is C++. I am using the latest version of LCM.

My messages are small (typically 50 bytes) but the frequency is pretty high (30 K msg/sec).

I notice missing messages on the receiver end. Although I am aware that delivery is not guaranteed with UDP, I am trying to see if I can stop losing messages.

I see no missing messages with throttled traffic (5K msg/sec). Messages start being dropped at 20 K msg/sec.

I am using a 12-core ubuntu dedicated server on the cloud (not a VPS or VM).

The CPU does not seem to be the bottleneck (usage is in the 10% range in htop).

I do not see dropped packets on the UDP stack (netstat -suna shows 0 receive buffer errors).

I have set this in /etc/sysctl.conf:

net.core.rmem_max=2097152 net.core.rmem_default=2097152

What could be the bottleneck?

My full-throttle bandwidth is 30 K x 50 bytes = 1.5 GB/s. I admit this is pretty high, but do not know which bottleneck I am hitting.

Is there some tuning that I can do to help?

Would I have better luck with a TCP framework such as RabbitMQ or ZeroMQ? (I'm not keen on rewriting my whole pipeline...)

Thanks for your help!

skdjfnd commented 2 years ago

I found the root cause of my issue, which was entirely unrelated to LCM. Closing this issue.