search

ros2 / rosbag2

Apache License 2.0
272 stars 245 forks source link

Performance Issues #1787

Open ben-aerovect opened 1 month ago

ben-aerovect commented 1 month ago

Description

Trying to record an MCAP bag of five 1080x720 30fps cameras and getting some performance issues. I'm on humble. I'm trying to troubleshoot whether this is an issue with rosbag2, or something else. Hardware wise, the system is on an i9-13900 and some generic nvme ssd. Using these commands fio --name TEST --eta-newline=5s --filename=fio-tempfile.dat --rw=write --size=500m --io_size=10g --blocksize=1024k --ioengine=libaio --fsync=10000 --iodepth=32 --direct=1 --numjobs=1 --runtime=60 --group_reporting.

dd if=/dev/zero of=/tmp/output conv=fdatasync bs=384k count=1k; rm -f /tmp/output

I"m getting around 620 MB/s write speed. However, it seems like the rosbag2 recording is capping out at bandwidth of 110-120MB/s. The bandwidth of each camera topic is around 70MB/s. Recording only 1 camera results in no throttling. Recording 2 cameras drops each to about 25hz, 3 drops to 17hz, ..., all 5 cameras is 10hz. Using compression or no compression doesn't make a difference, sqlite3 or mcap doesn't make a difference either. Seems like no matter what, I can't record more than 120MB/s of data without dropping frames despite the disk write speed being much faster. CPU utilization is under 40% on all threads while recording.

Is this a known limitation of rosbag2? Should I benchmark my disk differently?

Expected Behavior

No dropped frames until the hardware limitations (write speed) of the disk are being hit.

Actual Behavior

See above.

To Reproduce

** Steps to reproduce the behavior, e.g.

  1. Do something to generate 120+MB/s of data
  2. Run ros2 bag record camera1 camera2 ... -o bag

System (please complete the following information)

MichaelOrlov commented 1 month ago

@ben-aerovect Quick advise try to use and increase --max-cache-size parameter.

MichaelOrlov commented 1 month ago

There are no known issue in performance. I was able to record up to the 2.6 GBytes/Sec from 8 camears.

ben-aerovect commented 1 month ago

@ben-aerovect Quick advise try to use and increase --max-cache-size parameter.

--max-cache-size 104857600 didn't help. Can you please share the exact command and setup (ROS distro, DDS, camera hardware, etc) you used to get 2.6GB/s so I can try to reproduce it?

MichaelOrlov commented 1 month ago

@ben-aerovect See details in my blog post here https://www.apex.ai/post/improvements-in-data-recording-path. At that time rolling was matched to the Iron. To achieve a high throughput with a low CPU load, I used the zero-copy feature. In ROS 2, zero-copy for recording has been available since Jazzy's release. However, for your throughput, I think It should be enough to use non-zero copy transport and MCAP format. I would recommend using rosbag2_performance_benchmarking for rosbag2 performance evaluation on your setup. It will help to eliminate messages lost on a publisher's side and see how rosbag2 perform with different configurations.

ben-aerovect commented 1 month ago

@MichaelOrlov Thanks! I'll check those out and report back