problem displaying pointclouds that come by small packets at high rate

[bricerebsamen]

Hi

I am migrating from fuerte to hydro and encountered an issue with rviz in hydro. I am visualizing velodyne data, using @jack-oquin's driver. The default is to publish whole spins (large point cloud every 100ms), but you can choose to publish fractions of it at a time, so that you get more frequent and smaller pointclouds. I tried to publish one packet at a time, which corresponds to 384 points, or roughly 1 degree of data.

Then in rviz, I should be able to see the whole spin by setting the decay time to 0.1 (a revolution takes 100ms). And this works well in fuerte.

In hydro however, I see only 3 small fractions of the environment that are slowly rotating counterclockwise (the velodyne spins clockwise). See the attached screenshot

It's as if many packets are dropped. I tried setting the queue size to a large value (400 should be more than enough to hold 100ms of data) but it did not help.

how to reproduce: download the pcap file from here: http://driving.stanford.edu/data/sample_velodyne_data.pcap if you don't have a velodyne. Then replay it using velodyne_driver:

rosrun velodyne_driver velodyne_node _model:=64E_S2 _pcap:=vel.pcap
rosrun velodyne_pointcloud cloud_node

This should publish a pointcloud2 on /velodyne_points and should show up fine in rviz. then kill the driver and restart it with the npackets parameter set to 1:

rosrun velodyne_driver velodyne_node _model:=64E_S2 _npackets:=1

you should see the publish rate around 4700, and rviz should show something weird. Set npackets to 15 or more and it should be fine again.

[dgossow]

The pcap link gives me a 404.

[bricerebsamen]

right, I had to move it because it was eating up all my quota. Here is the new link: http://driving.stanford.edu/data/sample_velodyne_data.pcap I'll try to edit the original question.

[bricerebsamen]

No progress on this issue. Anything I can do to help?

[wjwwood]

I cannot do this line:

rosrun velodyne_driver velodyne_node model:=64ES2 pcap:=vel.pcap

After modifying velodyne_driver to catch exceptions, I find that this is the problem:

Character [6] is not valid as the first character in Graph Resource Name [64ES2].  Valid characters are a-z, A-Z, / and in some cases ~.

Though this might be a bug in roscpp. Either way, after putting the example into a launch file, I get this:

% roslaunch test.launch
... logging to /Users/william/.ros/log/04458bf5-62ab-11e3-bf85-542696cef915/roslaunch-dosa-84736.log
Checking log directory for disk usage. This may take awhile.
Press Ctrl-C to interrupt

started roslaunch server http://dosa:53168/

SUMMARY
========

PARAMETERS
 * /rosdistro
 * /rosversion
 * /velodyne_node/model
 * /velodyne_node/pcap

NODES
  /
    velodyne_node (velodyne_driver/velodyne_node)

ROS_MASTER_URI=http://localhost:11311

core service [/rosout] found
process[velodyne_node-1]: started with pid [84741]
[ERROR] [1386798055.927570000]: unknown Velodyne LIDAR model: 64ES2
[ INFO] [1386798055.929027000]: Velodyne HDL-64ES2 rotating at 600 RPM
[ INFO] [1386798055.930501000]: publishing 260 packets per scan
[ INFO] [1386798055.932913000]: expected frequency: 10.000 (Hz)
[ INFO] [1386798055.936545000]: Opening PCAP file "/Users/william/Downloads/sample_velodyne_data.pcap"
^C[velodyne_node-1] killing on exit
shutting down processing monitor...
... shutting down processing monitor complete
done

Are you sure 64ES2 is the correct model?

[wjwwood]

Sorry, the github markdown converted 64E_S2 into 64ES2.

[wjwwood]

I used this launch file:

<launch>
  <node pkg="velodyne_driver" name="velodyne_node" type="velodyne_node" output="screen">
    <param name="model" value="64E_S2" />
    <param name="npackets" value="1" />
    <param name="pcap" value="/path/to/sample_velodyne_data.pcap" />
  </node>
  <node pkg="velodyne_pointcloud" name="cloud_node" type="cloud_node" output="screen" />
</launch>

[wjwwood]

I can reproduce this:

https://vimeo.com/81664408

I think that there could be many causes for this performance hit, because many things have changed. The shaders used to render point clouds are now GLSL instead of the nVidia CG scripts. I am not sure, but I also think that the queuing of point clouds has been changed as well.

@hersh @dgossow Do you guys have any ideas/leads on what the most likely culprit here is?

[bricerebsamen]

@wjwwood what you show in your vimeo video is exactly the problem I'm experiencing. thanks for working on this.

[wjwwood]

No problem, specifically in the video I tried to point out that the fuerte rviz is saying it received 100's or 1000's of messages per second and the hydro rviz is reporting 10's of messages per second.

[wjwwood]

The rviz in groovy behaves the same as the rviz in hydro, so it was likely a change introduced between fuerte and groovy. I checked this to check and see if the change in tf between groovy and hydro could be the problem, that seems unlikely now.

[dgossow]

We changed how point clouds are rendered in hydro. Now, each laser scan gets it's own point cloud. Before, point clouds were basically generated in chunks. I guess that might have broken it. Also, the way how old point clouds are thrown out has changed. See https://github.com/ros-visualization/rviz/blob/hydro-devel/src/rviz/default_plugin/point_cloud_common.cpp#L532.

One question: does the frame rate drop significantly? If so, point clouds might get outdated while the processing takes place and never be displayed. See https://github.com/ros-visualization/rviz/blob/hydro-devel/src/rviz/default_plugin/point_cloud_common.cpp#L581

[dgossow]

I think I might have an idea what the problem is: RViz is keeping up a window relative to now(). If you have a window of 100ms and the latest message you have in your queue is already 90 ms old, you only display 10% old your clouds.

[bricerebsamen]

I checked that last point. It does not seem to be the case.

In PointCloudCommon::processMessage() I added some debug print to see what messages are received at what time.

ROS_INFO_STREAM("Received a cloud: stamp=" <<std::setprecision(16) <<cloud->header.stamp.toSec() <<", seq=" <<cloud->header.seq <<", frame_id=" <<cloud->header.frame_id <<", npoints=" <<(cloud->height*cloud->width));ROS_INFO_STREAM("Received a cloud: stamp=" <<std::setprecision(16) <<cloud->header.stamp.toSec() <<", seq=" <<cloud->header.seq <<", frame_id=" <<cloud->header.frame_id <<", npoints=" <<(cloud->height*cloud->width));

The result is that there is only a 10ms delay between the message stamp and now(). However, the message seq do not come right, some are skipped.

I compared with another node that is also subscribing from the same topic, and the seq number come in a neat sequence.

I tried playing back at a 10th of the real time speed and it did not change anything.

Also it does not seem that the frame rates drops, at least I don't notice it. Is there a way to measure it?

[wjwwood]

I haven't had any time to work on this, @bricerebsamen have you found a work around or learned anything new about the issue?

[bricerebsamen]

I have simple workarounds: set the driver to publish larger bundles of scans less frequently (i.e. a larger value of npackets). Or use an intermediate node that will aggregates scans, and have rviz subscribe to the output of that node. Nothing really helpful unfortunately.

[wjwwood]

Sorry, I haven't had any time for this, does anyone else have any time to look into this?

[y0grt]

Hi!

I'm experiencing the same problem with a Velodyne HDL-64E S3 in ros indigo (same packet rate that S2.1).

If try to visualize the pointcloud with the standard packet rate (providing a full scan per pointcloud) the image is flickering all time with a small black section spinning counterclockwise. With a high packet rate (npackets = 1 in the driver, providing a ros message per packet) I obtain the same result that Brice in his screenshot.

Any help?

Thanks for all.

[ffurrer]

I'm still encountering this in jade, for now, I'm just publishing all the different pointclouds as one (just for visualization in rviz).

[professionaljesus]

Having the same issue with VLP-16 and ros-kinetic, any new leads?

[rhaschke]

I looked at the issue in the latest melodic-branch of rviz. I can reproduce the issue using the velodyne pcap file, but I didn't see dropped messages in rviz: the message seq continuously grows when printed as suggested in https://github.com/ros-visualization/rviz/issues/689#issuecomment-30463690. As it is hard to understand, which messages are displayed and which are dropped, I modified the send_point_cloud_2.cpp test, which publishes now a grid of points in an incremental fashion: rosrun rviz send_point_cloud_2 10 There, increasing the decay timeout from 0.0 to 1.0 incrementally increases the number of shown points in a coherent fashion. Hence, I cannot see an issue in rviz. Did anybody ever consider that this might be a bug in the velodyne sources?