search

usdot-fhwa-stol / carma-platform

CARMA Platform is built on robot operating system (ROS) and utilizes open source software (OSS) that enables Cooperative Driving Automation (CDA) features to allow Automated Driving Systems to interact and cooperate with infrastructure and other vehicles through communication. Doxygen Source Code Documentation :
https://usdot-fhwa-stol.github.io/documentation/carma-platform/
400 stars 122 forks source link

rviz2 foxy is getting below 10fps when PCL map is loaded #2427

Open MishkaMN opened 1 month ago

MishkaMN commented 1 month ago

Summary

When loading point cloud map on ROS1 rviz (even when started inside the docker container as such above), the fps stays stable at 31fps. However, on rviz2, it comes down to 18-24 fps when static and zoomed in around the vehicle. And down to 6fps when the vehicle is moving (down to 6fps if zoomed out while static and the whole map is rendered on the view). Simulation PCs still work great at 31fps with PCL map loaded due to its fast processor/GPU (or might be due to different rmw of fastrtps).

I investigated this a bit more, and tried applying thishttps://github.com/ros2/rviz/pull/668 that is implemented for https://github.com/ros2/rviz/issues/670by building rviz2 from source, but it didn't help. I didn't put too much effort in making it more stable since

  1. we will upgrade to humble anyways and
  2. we have workaround of using ROS1 rviz if needed, and
  3. 6-8fps is not great, but still "doable". We can actually just turn off PCD and Lidar while carma is running which bumps the fps to 15+. This is because those are only needed during initialization at the moment.

So this issue is to improve the foxy version or verify it is not an issue and marked done when upgraded to humble.

Reasoning for new functionality

No response

MishkaMN commented 1 month ago

As part of the port, I tried testing following:

  1. running rolling and humble in docker images to display rviz2. However, despite the container has access to nvidia gpu. rviz2 was using the GPU in both cases. I am assuming I am configuring it wrong although I am running the container with all same configs ran by carma exec script, which works in foxy. I could be missing some environment variable that carma sets that I maybe missing. Since GPU is not usable, it is almost 1fps.
  2. I reduced the number of nodes that are running, it looks like it improves very marginally.
  3. Turning off route_marker somehow also improves the fps marginally.
  4. Tried using ROS_DOMAIN_ID=0 hoping to isolate if there is any other network issue for docker containers, but didn't affect (I was not successful in setting up the IPs without using host network though which might be avenue to look at)
  5. Tried using rmw_fastrtps_cpp instead of cyclonedds, it made it worse where average fps stayed the same, but every 4 second or so rviz display would freeze briefly then catch up when vehicle is running. John C made a point that he had trouble ending big data such as PCD map over rmw_fastrtps_cpp, so it is probably best to use cyclone.