search

ros / ros_comm

ROS communications-related packages, including core client libraries (roscpp, rospy, roslisp) and graph introspection tools (rostopic, rosnode, rosservice, rosparam).
http://wiki.ros.org/ros_comm
753 stars 911 forks source link

ROS messages published/recorded out of order for a few seconds #2336

Closed scott-robotics closed 1 year ago

scott-robotics commented 1 year ago

I'm not sure how best to describe this but after noticing very strange robot behavior, I looked into the message timestamps. I found that somehow the message is order is getting screw up - newer messages are getting interleaved with older ones. The timestamps the messages are reporting appear to be correct (monotonic) but the seq and the values are out of order, alternating between old and new messages. This also happens on other topics, like wheel odometry. I've shown the gyro topic reported from the realsense node. Note also the rogue timestamp in the lower plot.

Any insight or avenues to debugging would be very appreciated!

ROS noetic Ubuntu 20.04 - 5.11.0-46-generic x86 - Intel i5

image

scott-robotics commented 1 year ago

If anyone comes across something similar, the root cause was the system clock synchronizing over the internet, adjusting itself by a few seconds each time. Any state estimation requiring precise timestamping goes haywire. Symptoms went away after disabling online time clock sync