ros-drivers / velodyne

ROS support for Velodyne 3D LIDARs
http://ros.org/wiki/velodyne
Other
652 stars 651 forks source link

~1x per hour the diagnostics complains with "No data since last update." #474

Open Rayman opened 2 years ago

Rayman commented 2 years ago

Please complete the following information:

Describe the bug Randomly, approximatly 1x per hour the velodyne driver gives out a single WARN diagnostics message. The point cloud stream is still good. I have a feeling this is a threading/locking issue with the diagnostics updater.

To Reproduce

  1. Run the driver
  2. Record a bagfile for 1 hour or more
  3. Run the following command to figure out if the bug happens rostopic echo -b file.bag /diagnostics/status[0]/message | grep '\w'
  4. In plotjuggler, you can plot Events in window to visually see when the bug occurs.

Expected behavior The point cloud data stream is good, so I expect the diagnostics to always send out OK.

Additional context Diagnostics visualized in plotjuggler. I wrote an external diagnostics_updater that listends to the pointcloud data and it never reports a problem. Both diagnostics are visualized below. image

This is the full diagnostics message that is produced:

level: 1
name: "velodyne_nodelet_manager_driver: velodyne_packets topic status"
message: "No data since last update."
hardware_id: "Velodyne VLP-16"
values: 
  - 
    key: "Events in window"
    value: "90"
  - 
    key: "Events since startup"
    value: "35120"
  - 
    key: "Duration of window (s)"
    value: "9.077190"
  - 
    key: "Actual frequency (Hz)"
    value: "9.914963"
  - 
    key: "Target frequency (Hz)"
    value: "9.921053"
  - 
    key: "Minimum acceptable frequency (Hz)"
    value: "4.960526"
  - 
    key: "Maximum acceptable frequency (Hz)"
    value: "14.881579"
  - 
    key: "Earliest timestamp delay"
    value: "No data"
  - 
    key: "Latest timestamp delay"
    value: "No data"
  - 
    key: "Earliest acceptable timestamp delay"
    value: "-1.000000"
  - 
    key: "Latest acceptable timestamp delay"
    value: "5.000000"
  - 
    key: "Late diagnostic update count"
    value: "0"
  - 
    key: "Early diagnostic update count"
    value: "0"
  - 
    key: "Zero seen diagnostic update count"
    value: "0"
---
JWhitleyWork commented 1 year ago

Sorry it took me so long to get to this. If this issue persists, please try to verify if the drop in data are due to the data stream from the sensor being interrupted (e.g. record a pcap file during a drop-out) or from the driver. Having information about system memory utilization would help too as this could be a memory leak.

Rayman commented 1 year ago

The strange thing is that the pointcloud data is still arriving from the ROS driver at the expected rate. The problem is purely in the diagnostics message.