yeff-fresh
commented
6 months ago Update: I took the TB4 home, so I could use my home wireless network with internet access. The RaspPi time-sync'd with whatever ntp time server it uses and the Create3 time-sync'd with the RaspPi. So it appeared that the RaspPi, the Create3, and my PC were all in the "same time". I still did not see various Create3 topics on my PC.
I was able to run SLAM on the RaspPi and enough topics appeared on my PC that I could actually run RVIZ and see what the T4 saw as it drove around my house. However, the map was very low resolution. (I was able to save the map, however)
I was also able to run localization and nav2 on the RaspPi and again run RVIZ on my PC. However, there were still several topics missing and I wasn't able to see much from the scan).
So I still have ISSUE #2 from above where I can't see all the Create3 topics on my PC even though the three systems appear to be time-sync'd.
I also have another question which is: How can I configure the RaspPi to time-sync off a local ntp server, versus whatever it is currently using? There doesn't appear to be a /etc/ntp.conf file on the RaspPi - is it elsewhere?
I'll look around some more later today, but any help would be appreciated. Thanks!
hilary-luo
(ETA: Added a summary at the end of this original description of the Issue with what the various problems were, and how they were solved.)
Robot Model
Turtlebot4 Standard
ROS distro
Humble
Networking Configuration
Simple Discovery
OS
Ubuntu 22.04
Built from source or installed?
Installed
Package version
Create 3 is using firmware version H.2.4 (updated through web server) from Create 3 Releases (https://iroboteducation.github.io/create3_docs/releases/overview/) Raspberry Pi 4 is using code image Humble 1.0.0 from TurtleBot4 Humble page (https://turtlebot.github.io/turtlebot4-user-manual/changelogs/humble.html) I have done sudo apt update && sudo apt upgrade on the RaspPi4 PC is running Ubuntu 22.04 and ROS2 Humble
Type of issue
Navigation (SLAM, Nav2 etc.)
Expected behaviour
I can run RVIZ and localization and mapping on my PC and see a clean map in RVIZ with the localization and navigation indicators and can see all topics and messages on my PC (and even generate a map as per UserManual -> TUTORIALS -> 4. Generating a map)
Actual behaviour
RVIZ on PC displays error that Map frame is missing /odom topic not on PC (and others also missing), but it is on RaspPi4 /tf does not comtain odom->base_frame transform (other transforms missing too) but it is on RaspPi4
In debugging, went down the road of checking ntp time sync and did get Create3 and RaspPi4 sync'd with correct time (same as PC). But setup is non-persistent and will not work if TB4 not on internet-connected network (of course), and still don't see topics even if everyone is on same internet-connected network.
Error messages
To Reproduce
TB4 on wireless network PC on same wireless network Start TB4 Run RVIZ/localization/navigation on PC
Other notes
(Here comes all the gruesome details)
I wanted to have my Turtlebot4 and my Ubuntu PC on the same wireless network and be able to run RVIZ, localization, and navigation on my PC for both making a map and for navigation. (As per Turtlebot4 User Manual, TUTORIALS section, "4. Generating a map" and "5. Navigation"
I first tried putting my PC on the "Turtlebot4" network - the default the T4 starts with as an access point. This didn't work - RVIZ showed errors about Map frame not available and I never saw the scan results in RVIZ.
I switched to putting my T4 on a wireless network. My site wireless is very picky about what it allows, so I switched to my home wireless network/internet. I was able to put the T4 on that wireless network, but things sitll didn't work - RVIZ showed errors about Map frame.
Poking around ROS2 on my PC and on the T4 RaspPi, I noticed there were topics on the RaspPi that were not on my PC. Important topics like /odom, and in fact all the missing topics appeared to be ones that originate with the Create3.
Searching through the Turtlebot4 Issues board found Issue #300 which seemed completely relevant to my situation. The problem in Issue #300 was that the RaspPi and the Create3 were not time synced. Looking at ROS topics, I realized all the packets on /odom were significantly behind the "current time" (by 12 days). Also transforms on /tf from odom->base_link (for example) were on the RaspPi with old timestamps. The /odom topic was not visible on my Ubuntu PC, nor were the odom->base_link transforms visible in /tf on my PC. I also saw the line in the Create3 LOGS about ntpd trying to timesync with irobot.pool.ntp.org servers.
On the Issues board, I found Issue #216 which showed how to configure the Create3 ntp.conf through the WebUI. Doing this did seem to get the two systems in time sync, based on looking at the Create3 time in the WebUI and the RaspPi time in a shell (after ssh to the RaspPi).
After a lot of fiddling around I also figured out that if I put the RaspPi onto a different wireless network, say its own wireless network or a network that didn't have internet access, the RaspPi time reverted to a time in the past (specifically, the day I installed the 1.0.0 image). Then the Create3 would time sync to the RaspPi and thus my laptop (once on the same wireless network) would not see any ROS2 information (topics or messages). From this I realized that the RaspPi was timesync-ing to an internet-based time server and thus would revert time when not on a wireless network with internet access.
ISSUE #1: I can't guarantee that the Turtlebot4 will always be on a wireless network that has internet access.
My solution for this would be to install a small real-time clock module on the RaspPi (like one from AdaFruit or MakerFocus. I'm wondering if this is something that would be recommended for the Turtlebot4 by Clearpath, and if there is a recommended add-on. I haven't yet exposed the board to see if the needed pins are available.
ISSUE #2: Even when I have the T4 and my PC on a wireless network with internet access, I still don't see all the topics on my PC (or all the transforms in /tf). I did confirm that the time does appear to be in sync between RaspPi, Create3, and my PC. I even confirmed that some messages on /tf are on /tf on both the RaspPI and my PC.
I'm not as sure how to continue to debug this one. I'm wondering if it could be a routing issue between my PC and the Create3 (through the RaspPi). As mentioned in the User Manual, SETUP->4.DiscoveryServer->UserPC However, the appearance of some Create3 ROS2 topics (like /cmd_vel) would imply that perhaps routing is okay? Any ideas here would be welcome.
Thanks!
Post-Fixes Summary
There were several issues involved in this:
The RaspPi wants to time-sync with a time server on the internet. So it needs to be on a wireless network that has Internet access. Otherwise, the time will remain at the date when the code image was created. (It could be that adding a RealTimeClock module to the Raspberry Pi would solve this issue, as the Pi would maintain time over reboots. I have not tried this yet.). Note that the Create3 also tries to time-sync with an Internet time server, but this can be changed using the Create3 WebUI. If ROS2 messages have wildly different (old) timestamps, they can be ignored.
The RaspPi and the Create3 BOTH need to be on the same wireless network. They are connected via a USB-Ethernet adaptor, but if other computers on the wireless network need to see ROS2 topics and messages then ensure both RaspPi and Create3 are on that wireless network. The RaspPi wireless network is configured with "turtlebot4-setup" (run in a shell on the RaspPi), the Create3 wireless network is configured with the Create3 WebUI.
Ensure that all ROS2 systems are using the same DDS middleware. By default, ROS2 Humble uses FastRTPS. But it is easy to set the middleware to CycloneDDS. To confirm what a ROS2 system is using for middleware, check the value of the shell variable "RMW_IMPLEMENTATION". A value of "rmw_fastrtps_cpp" is (of course) FastRTPS, a value of "rmw_cyclonedds_cpp" is of course Cyclone DDS. This can be set at the command line or in the shell configuration (like .bashrc). I believe it can also be set for the Create3 in the WebUI (I haven't done this yet).