SLAM not working from PC - rplidar_link discarded

SamanthaSmith04 commented 4 months ago

Robot Model

Turtlebot4 Standard

ROS distro

Humble

Networking Configuration

Simple Discovery

OS

Ubuntu 22.04

Built from source or installed?

Installed

Package version

samubuntu@samubuntu:~$ dpkg -s ros-$ROS_DISTRO-turtlebot4-navigation Package: ros-humble-turtlebot4-navigation Status: install ok installed Priority: optional Section: misc Installed-Size: 2470 Maintainer: rkreinin rkreinin@clearpathrobotics.com Architecture: arm64 Version: 1.0.4-1jammy.20240530.144553 Depends: ros-humble-nav2-bringup, ros-humble-nav2-simple-commander, ros-humble-slam-toolbox, ros-humble-ros-workspace Description: Turtlebot4 Navigation

Type of issue

Navigation (SLAM, Nav2 etc.)

Expected behaviour

The updated map should appear in RViz along with LiDAR data when both ros2 launch turtlebot4_navigation slam.launch.py and ros2 launch turtlebot4_viz view_robot.launch.py are run on the PC.

I can get the map to work when running ros2 launch turtlebot4_navigation slam.launch.py through ssh on the raspberry pi and ros2 launch turtlebot4_viz view_robot.launch.py on the PC.

Actual behaviour

A static image of the map is shown only, no lidar data is displayed, and most of the tf and robotmodel information is not connected.

Instead of registering the Lidar sensor, it just keeps printing that the message filter is dropping the rplidar_link frame

Error messages

Running SLAM from PC
samubuntu@samubuntu:~$ ros2 launch turtlebot4_navigation slam.launch.py 
[INFO] [launch]: All log files can be found below /home/samubuntu/.ros/log/2024-07-08-11-16-16-363943-samubuntu-3634
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [sync_slam_toolbox_node-1]: process started with pid [3635]
[sync_slam_toolbox_node-1] [INFO] [1720451776.448386010] [slam_toolbox]: Node using stack size 40000000
[sync_slam_toolbox_node-1] [INFO] [1720451776.483089094] [slam_toolbox]: Using solver plugin solver_plugins::CeresSolver
[sync_slam_toolbox_node-1] [INFO] [1720451776.483291135] [slam_toolbox]: CeresSolver: Using SCHUR_JACOBI preconditioner.
[sync_slam_toolbox_node-1] [INFO] [1720451776.912736969] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866386.732 for reason 'discarding message because the queue is full'
[sync_slam_toolbox_node-1] [INFO] [1720451777.112955011] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866386.855 for reason 'the timestamp on the message is earlier than all the data in the transform cache'
[sync_slam_toolbox_node-1] [INFO] [1720451777.183319427] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866386.991 for reason 'discarding message because the queue is full'
[sync_slam_toolbox_node-1] [INFO] [1720451777.317997177] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866387.126 for reason 'discarding message because the queue is full'
[sync_slam_toolbox_node-1] [INFO] [1720451777.441654719] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866387.260 for reason 'discarding message because the queue is full'
[sync_slam_toolbox_node-1] [INFO] [1720451777.639544636] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866387.395 for reason 'discarding message because the queue is full'
[sync_slam_toolbox_node-1] [INFO] [1720451777.711557386] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866387.519 for reason 'discarding message because the queue is full'
[sync_slam_toolbox_node-1] [INFO] [1720451777.847515761] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866387.654 for reason 'discarding message because the queue is full'
^C[WARNING] [launch]: user interrupted with ctrl-c (SIGINT)
[sync_slam_toolbox_node-1] [INFO] [1720451777.910908011] [rclcpp]: signal_handler(signum=2)
[INFO] [sync_slam_toolbox_node-1]: process has finished cleanly [pid 3635]
samubuntu@samubuntu:~$

To Reproduce

On PC:

ros2 launch turtlebot4_navigation slam.launch.py

ros2 launch turtlebot4_viz view_robot.launch.py

Other notes

We have a router set up to run an isolated network without Internet access, since we are unable to work with the robot on our university WiFi.

Viewing topics

I can view Create3 topics from both my PC and while ssh'd into the turtlebot:

Turtlebot4 topic list: 

ubuntu@ubuntu:\~$ ros2 topic list /battery_state /cliff_intensity /cmd_audio /cmd_lightring /cmd_vel /diagnostics /diagnostics_agg /diagnostics_toplevel_state /dock_status /function_calls /hazard_detection /hmi/buttons /hmi/display /hmi/display/message /hmi/led /imu /interface_buttons /ip /ir_intensity /ir_opcode /joint_states /joy /joy/set_feedback /kidnap_status /mobility_monitor/transition_event /mouse /oakd/rgb/preview/image_raw /odom /parameter_events /robot_description /robot_state/transition_event /rosout /scan /slip_status /static_transform/transition_event /stop_status /tf /tf_static /wheel_status /wheel_ticks /wheel_vels

PC topic list:

samubuntu@samubuntu:~$ ros2 topic list /battery_state /cliff_intensity /cmd_audio /cmd_lightring /cmd_vel /diagnostics /diagnostics_agg /diagnostics_toplevel_state /dock_status /function_calls /hazard_detection /hmi/buttons /hmi/display /hmi/display/message /hmi/led /imu /interface_buttons /ip /ir_intensity /ir_opcode /joint_states /joy /joy/set_feedback /kidnap_status /mobility_monitor/transition_event /mouse /oakd/rgb/preview/camera_info /oakd/rgb/preview/image_raw /odom /parameter_events /robot_description /robot_state/transition_event /rosout /scan /slip_status /static_transform/transition_event /stop_status /tf /tf_static /wheel_status /wheel_ticks /wheel_vels

Verified that I can echo Create3 topics

samubuntu@samubuntu:~$ ros2 topic echo /odom header: stamp: sec: 1719866066 nanosec: 477515955 frame_id: odom child_frame_id: base_link pose: pose: position: x: -0.0004107786517124623 y: -3.549097527866252e-05 z: -1.1397002708690707e-05 orientation: x: -0.007184548769146204 y: -0.014398712664842606 z: 0.00022355528199113905 w: 0.9998704791069031 covariance:

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0 twist: twist: linear: x: 0.0 y: 0.0 z: 0.0 angular: x: 0.0 y: -1.612569278108822e-08 z: 0.0 covariance:
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0 …

Running SLAM on turtlebot through SSH

We also get very similar output when running SLAM from the turtlebot, but it does connect properly after a few seconds:

Turtlebot4 running SLAM:

ubuntu@ubuntu:~$ ros2 launch turtlebot4_navigation slam.launch.py [INFO] [launch]: All log files can be found below /home/ubuntu/.ros/log/2024-07-01-20-41-18-548417-ubuntu-1623 [INFO] [launch]: Default logging verbosity is set to INFO [INFO] [sync_slam_toolbox_node-1]: process started with pid [1624] [sync_slam_toolbox_node-1] [INFO] [1719866479.161202370] [slam_toolbox]: Node using stack size 40000000 [sync_slam_toolbox_node-1] [INFO] [1719866479.766219844] [slam_toolbox]: Using solver plugin solver_plugins::CeresSolver [sync_slam_toolbox_node-1] [INFO] [1719866479.771028060] [slam_toolbox]: CeresSolver: Using SCHUR_JACOBI preconditioner. [sync_slam_toolbox_node-1] [INFO] [1719866479.904952193] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866479.635 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866480.039628467] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866479.771 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866480.174491869] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866479.905 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866480.310197134] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866480.041 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866480.432755407] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866480.175 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866480.567474885] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866480.310 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866480.703460906] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866480.433 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866480.838006589] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866480.568 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866480.973673947] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866480.704 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866481.095743855] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866480.838 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866481.230815366] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866480.974 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866481.366307799] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866481.096 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866481.501812399] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866481.231 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866481.636620913] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866481.367 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866481.759066095] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866481.502 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866481.894154458] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866481.637 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866482.029381894] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866481.759 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866482.164271055] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866481.894 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866482.300640369] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866482.029 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866482.434570447] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866482.165 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866482.557655437] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866482.301 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866482.692494655] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866482.435 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866482.827504741] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866482.558 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866482.962973915] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866482.693 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866483.098270924] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866482.828 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866483.220610126] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866482.963 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866483.355588453] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866483.099 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866483.490889166] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866483.221 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866483.625856382] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866483.356 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866483.761243446] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866483.492 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866483.896142903] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866483.626 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866484.019036692] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866483.762 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866484.154446126] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866483.896 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866484.289666599] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866484.019 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] [INFO] [1719866484.424249430] [slam_toolbox]: Message Filter dropping message: frame 'rplidar_link' at time 1719866484.155 for reason 'discarding message because the queue is full' [sync_slam_toolbox_node-1] Registering sensor: [Custom Described Lidar]

Create 3 logs

Here is my Create3 (H2.6) log too: messages(1).txt Another notable concern is that the date/time is incorrect on the create3/raspberry pi4

Let me know if there is any other information that I should provide!

RustyCPR commented 3 months ago

Hi @SamanthaSmith04, thanks for your patience, I was working with our Engineering team on your concerns, there are a few items we would like you to try:

Setup the NTP config on the Create3 so that the Create3 time syncs with the RPi instead of attempting to sync over the internet - see instructions in ticket #216
The default NTP Ubuntu time syncing systems (timedatectl and timesyncd) are insufficient when you need to offer time syncing services to other devices on the network. See these instructions: https://ubuntu.com/server/docs/how-to-serve-the-network-time-protocol-with-chrony. Please follow the instructions up until and including the section that is "Enable serving the Network Time Protocol". Specifically you'll need to install chrony and then "allow" the RPi access to the server . You also need to make sure that you restart chrony after making any changes to the file (sudo systemctl restart chrony.service). As an additional note make sure that this device has a static IP so that the RPi can always find it at a given IP address.
You'll need to redirect the RPi to sync with the new NTP server that you've set up. Open the /etc/chrony.conf file and add a new server line similar to the one you've used for the Create3 but changing the IP address to point at the NTP server set up in the previous step. This is what will tell the RPi to sync to your computer and then you can verify that after they have restarted chrony that you are able to sync properly.

Any questions please let me know, Rusty

RustyCPR commented 2 months ago

Hi @SamanthaSmith04 any progress or updates?

RustyCPR commented 1 month ago

Closing due to no activity.

turtlebot / turtlebot4