Turtlebot4 Navigation Example Error "frame 'rplidar_link' at time 1709940493.141 for reason 'the timestamp on the message is earlier than all the data in the transform cache'"

Christian-Prather commented 8 months ago

Robot Model

Turtlebot4 Standard

ROS distro

Humble

Networking Configuration

Simple Discovery

OS

Ubuntu 22.04

Built from source or installed?

Installed

Package version

Package: ros-humble-turtlebot4-navigation Status: install ok installed Priority: optional Section: misc Installed-Size: 2470 Maintainer: rkreinin rkreinin@clearpathrobotics.com Architecture: amd64 Version: 1.0.4-1jammy.20240217.111144 Depends: ros-humble-nav2-bringup, ros-humble-nav2-simple-commander, ros-humble-slam-toolbox, ros-humble-ros-workspace Description: Turtlebot4 Navigation

Type of issue

Navigation (SLAM, Nav2 etc.)

Expected behaviour

The navigation tutorial to produce a map with lidar scan points overlay and a functional navigation path.

Actual behaviour

When following the navigation tutorial https://turtlebot.github.io/turtlebot4-user-manual/tutorials/navigation.html the first command that is ask you to run ros2 launch turtlebot4_navigation localization.launch.py map:=office.yaml runs but produced the following error that seems to be causing failure in subsequent steps.

Error: [amcl-2] [INFO] [1709940494.277377550] [amcl]: Message Filter dropping message: frame 'rplidar_link' at time 1709940493.141 for reason 'the timestamp on the message is earlier than all the data in the transform cache'

Error messages

[amcl-2] [INFO] [1709940494.277377550] [amcl]: Message Filter dropping message: frame 'rplidar_link' at time 1709940493.141 for reason 'the timestamp on the message is earlier than all the data in the transform cache'

To Reproduce

ros2 launch turtlebot4_navigation localization.launch.py map:=office.yaml

Other notes

I have looked and this seems to be a common error in recent months. I attempted all solutions recommended which include updating the turtlebot and my system, moving to cyclonedds, and attempting to sync the Create3 NTP server.

Any help would be appreciated as this is a complete blocker and renders our turtlebot useless.

smatarCPR commented 8 months ago

Hello @Christian-Prather, Thank you for reaching out on the official Turtlebot 4 Github page. I am very sorry to hear that you are encountering an issue. Out of curiosity, when you attempted to sync the Create3's NTP server, were you successful? Was there an error output?

Are using an internet enable network for the Raspberry Pi to be able to sync it's time? Is your computer connected to the same network and syncing its time using the network?

Would it be possible to try and use timesyncd to sync the time between the Raspberry Pi and your computer (https://ubuntu.com/server/docs/use-timedatectl-and-timesyncd).

Looking forward to hearing from you soon. Best Regards, Saif

Christian-Prather commented 8 months ago

Hello @smatarCPR To try and answer your questions the pi is attached to internet enabled wifi and the NTP I thought did not have errors however I have attached the latest logs and latest ntp config. There error is very persistent. ntp-config

create3-logs.txt

Christian-Prather commented 8 months ago

Its also worth noting that since I saw no improvement with cyclone I rolled back to default fast_dds

rupak-d commented 8 months ago

I have a similar timestamp issue. Using the 1.0.4 version ros-humble-turtlebot4-navigation package.

alexern14 commented 8 months ago

I am also having an issue similar to this. I get the same error that the lidar message time is earlier than all the data in the transform cache.

Christian-Prather commented 8 months ago

Additional update I have changed over the system and my laptop to use the discovery server and while the topics seem to show up more reliably on my laptop the same error reported occurs when running the example localization node.

hilary-luo commented 8 months ago

Since this seems to be a hot topic right now I am going to try and drop a bunch of general information here that I am hoping will be helpful to a number of you.

If you are running simulation then stop here, none of the below applies and your issue is likely on how you are launching the nodes (sim time vs real time). If you are using a physical robot then continue:

Both the Raspberry Pi and the Create3 drive base will sync their time with the internet by default. So there are a few different configurations and what could be causing your issues:

Simple Discovery - As long as your network is connected to the internet then all you should need to do is to make sure that your laptop is also syncing time with the internet and these should all match (for default ubuntu you can check using timedatectl status - this also works on the raspberry pi). If you still seem to be seeing issues then it could be helpful to reboot the robot and your PC and potentially restart the ntpd from the create3 webserver. Similarly try forcing a time sync on your laptop. (More details on how to check last time sync etc on ubuntu here -> it isn't sufficient to visually check and say that the time on your laptop looks right, they need to be the same )
Discovery Server - As long as your network is connected to the internet then all you should need to do is to make sure that your laptop is also syncing time with the internet (see the command to check this in option 1) AND redirect the create3 drive base to sync time with the raspberry pi (instructions here)
If your system is not connected to the internet then you will need to modify the timesyncing on both the create3 and raspberry pi. If you are using discovery server then still point the create3 at the raspberry pi (instructions here) and then point the raspberry pi at an NTP server on your PC. If it is simple discovery you can still do that or point both at your own NTP server.

A final note: If you look at what node and what topic are shown in the error then that will point you towards which devices are out of sync. For example, if you are running navigation on your laptop and that is where you are seeing the error pop up (the amcl node is part of the nav packages) then that tells you that your laptop is one of the pieces that is not in sync. Then you can look at the topic that it is not in sync with, and which device published that topic. In the case of the lidar, it is published by the Raspberry Pi. So as much as it is good to make sure that the create3 is in sync, that is not causing the particular issue being presented.

This behaviour should not change based on which DDS you select or which version of packages that you run. If you want to understand this concept better I would encourage you to look up NTP. Nothing about how the time is getting synced is ROS 2 specific, it is all built on existing packages and protocols. ROS just happens to care if the messages are out of sync.

smatarCPR commented 6 months ago

Hello, This issue is being closed due to inactivity. If you are still experiencing the issue, feel free to reopen this ticket when you are ready to continue the troubleshooting process. Best Regards, Saif

turtlebot / turtlebot4