Open lukeopteran opened 10 months ago
Hello! I haven't seen this behavior before, but I will try to replicate it. Can you tell me a little more about your RMW setup? Are you using Fast DDS or Cyclone and do you have a custom XML configuration on the Pi?
I believe that error happens when Cyclone DDS has trouble due to a network change, so it is quite possible that log message is an indicator. Is something changing on your network? The log seems to indicate that your robot is losing a network connection. In my experience, Cyclone DDS is quite sensitive to changes in network settings and does not recover well when an interface goes down.
Thanks for looking at this, we have lost the messaging with the robot under both Cyclone and FastRTPS
Under Cyclone we run a custom configuration on the Pi and on the robot, to specify the network interface and to prevent additional routing as we have found this give greater stability.
@shamlian I agree on cyclone and network adapters, this is why we rarely use it on the robot wifi side and stick to the usb0. However, when the wifi is working we get more reliable performance than FastRTPS. It would be good if we could disable the robot Wifi, only have it enable for hot-spotting....
Robot RMW config override: Galactic (5.2):
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain id="any">
<General>
<NetworkInterfaceAddress>usb0</NetworkInterfaceAddress>
<AllowMulticast>true</AllowMulticast>
<DontRoute>true</DontRoute>
<MaxMessageSize>65500B</MaxMessageSize>
</General>
</Domain>
</CycloneDDS>
Humble (2.2)
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain id="any">
<General>
<Interfaces>
<NetworkInterface name="usb0" />
<!--<NetworkInterface name="wlan0" />-->
</Interfaces>
<AllowMulticast>true</AllowMulticast>
<MaxMessageSize>65500B</MaxMessageSize>
<DontRoute>true</DontRoute>
</General>
</Domain>
</CycloneDDS>
On the Pi we have two configurations to block the robot from excess ROS2 traffic, so the pi to robot side has: Galactic
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain id="any">
<General>
<NetworkInterfaceAddress>usb0</NetworkInterfaceAddress>
<AllowMulticast>true</AllowMulticast>
<MaxMessageSize>65500B</MaxMessageSize>
<DontRoute>true</DontRoute>
</General>
</Domain>
</CycloneDDS>
Pi Humble cyclone cfg
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain id="any">
<General>
<Interfaces>
<NetworkInterface name="usb0"/>
</Interfaces>
<AllowMulticast>true</AllowMulticast>
<MaxMessageSize>65500B</MaxMessageSize>
<DontRoute>true</DontRoute>
</General>
</Domain>
</CycloneDDS>
Thanks
Hi @lukeopteran! I've run some tests and re-examined your logs. I was not able to replicate what you are seeing by just letting it sit idle for several hours. However, I was able to replicate the error you're seeing in the logs by removing the usb0 IP assignment from the netplan yaml on the Pi, applying the new netplan, restarting the Pi, and adding back and re-applying the usb0 settings. I ran this check to see if a disruption in that interface would cause an issue, and it did. My suspicion is that you may have a faulty USB-C connection. Are you able to test with a new cable and then upload a full copy of your logs? I'd love to capture the logs particularly right before you start getting the failed with retcode -1 errors.
Just to add to this -- it would be extremely helpful if you could share logs from the robot when this problem begins to happen, so we can try to reproduce the problem here. You've provided a snippet, but having the context and the event would be extremely helpful. If you are uncomfortable uploading the logs here, you can email them to education@irobot.com and they will make their way to the Create 3 team.
We have multiple robots with this setup, with different usbc leads, I don't think we have a physical poor connection. It also happens after a while when the robots aren't moving.
Here is a log file from just now, a reboot fixed it: messages no ros2 topics until reboot.txt
Thanks for the update. It looks like these logs don't capture the point at which the connection stopped working, which may be hard to do with your setup since it is hard to predict exactly when it goes down.
I've run a series of tests over the last 48 hours and am still unable to replicate the issue you are seeing. The Create 3 robot provided a topic list after 1 hour, 2 hours, and a full 24 hours with no reboot. Based on the information you've provided, there are two differences in our set ups. First, I factory reset my Create 3 robot to stop it from attempting to join a Wi-Fi network. The second is that I am not using any custom xml profiles for either the Create or the Pi. I'm going to run some tests with your settings now, but while I do, it would be helpful if you could attempt to replicate my set up so we can try to further isolate the root of the issue.
Edit: I should note I've been testing using Galactic and 20.04 since it sounded like your issues were persistent across both Humble and Galactic, but let me know if you're seeing more on one distribution vs another.
Hi again, I ran another overnight test replicating all of the settings you have provided, and I'm still able to receive a topic list after 12-18 hours of inactivity. Since I was only able to replicate the errors you are seeing in the logs by disrupting or altering the usb0 interface, I still suspect this is the cause. If you have tried different cables, then my next thought would be it is some sort of OS issue that is putting that interface to sleep on the Pi after a certain period of time. Can you please provide additional details on how you've configured your Ubuntu image on your Pi?
Had some more network over USB comms failures, I got a log with the failure and after when reboot and working again.
Just restarting the application didn't work.
Running latest Humble H2.6FW and cyclone custom RMW config
Thanks Luke
after full restart working again.txt usb_comms_fail.txt
Cyclone RMW config:
<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
<Domain id="any">
<General>
<Interfaces>
<NetworkInterface name="usb0" />
</Interfaces>
<AllowMulticast>true</AllowMulticast>
<DontRoute>true</DontRoute>
<MaxMessageSize>65500B</MaxMessageSize>
</General>
</Domain>
</CycloneDDS>
RPi using netplan cfg
# This file is generated from information provided by the datasource. Changes
# to it will not persist across an instance reboot. To disable cloud-init's
# network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
version: 2
renderer: networkd
ethernets:
usb0:
# To create 3
dhcp4: false
dhcp6: false
optional: true
addresses: [192.168.186.1/24]
eth0:
# Main ROS2 side
dhcp4: false
dhcp6: false
addresses: [192.168.9.99/24]
gateway4: 192.168.9.1
optional: true
nameservers:
addresses: [8.8.8.8, 8.8.4.4]
I'm wondering if there is a problem with the physical link. How well is your cable strain-relieved? Could it be bouncing around? Is it in good shape? Also, on the Pi, are there any interesting messages in dmesg
or /var/log/messages
when this happens?
The robot wasn't moving when the fault occurred, will check the logging
It has done it again, the robot is sat completely still, but has been on (charging) for a good while (>5 hours). I did software restart of the Pi at one point a while ago.
I get no ROS2 messages, but I can get to the Create3 web UI etc fine through the same network route (Pi > usb0)...
Here are any messages in /var/log relating to the usb0 connection
boot.log:[ 18.185007] cloud-init[863]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | e2:d5:92:d4:cf:90 |
boot.log:[ 18.185183] cloud-init[863]: ci-info: | usb0 | True | fe80::e0d5:92ff:fed4:cf90/64 | . | link | e2:d5:92:d4:cf:90 |
boot.log:[ 18.186482] cloud-init[863]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log:[ 18.187622] cloud-init[863]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log:[ 18.187802] cloud-init[863]: ci-info: | 1 | local | :: | usb0 | U |
boot.log:[ 18.188122] cloud-init[863]: ci-info: | 2 | multicast | :: | usb0 | U |
kern.log:Nov 21 21:10:38 irobot37 kernel: [ 23.668504] IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
kern.log:Nov 21 21:10:39 irobot37 kernel: [ 23.903330] IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
boot.log.5:[ 18.304626] cloud-init[861]: ci-info: | usb0 | False | . | . | . | 96:ab:29:04:c6:6e |
boot.log.5:[ 17.899344] cloud-init[868]: ci-info: | usb0 | False | . | . | . | be:30:19:4b:31:f4 |
boot.log.5:[ 18.233844] cloud-init[871]: ci-info: | usb0 | False | . | . | . | 22:5d:43:27:8d:f0 |
syslog:Apr 15 12:25:11 irobot37 avahi-daemon[891]: Leaving mDNS multicast group on interface usb0.IPv6 with address fe80::9cf9:aeff:fe42:f1e9.
syslog:Apr 15 12:25:11 irobot37 avahi-daemon[891]: Leaving mDNS multicast group on interface usb0.IPv4 with address 192.168.186.1.
syslog:Nov 21 21:10:34 irobot37 systemd-udevd[485]: usb0: Could not generate persistent MAC: No data available
syslog:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: IPv6 successfully enabled
syslog:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: Link UP
syslog:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: Gained carrier
syslog:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: Gained IPv6LL
syslog:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | e2:d5:92:d4:cf:90 |
syslog:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | usb0 | True | fe80::e0d5:92ff:fed4:cf90/64 | . | link | e2:d5:92:d4:cf:90 |
syslog:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
syslog:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
syslog:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | 1 | local | :: | usb0 | U |
syslog:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | 2 | multicast | :: | usb0 | U |
syslog:Nov 21 21:10:35 irobot37 avahi-daemon[891]: Joining mDNS multicast group on interface usb0.IPv6 with address fe80::e0d5:92ff:fed4:cf90.
syslog:Nov 21 21:10:35 irobot37 avahi-daemon[891]: New relevant interface usb0.IPv6 for mDNS.
syslog:Nov 21 21:10:35 irobot37 avahi-daemon[891]: Joining mDNS multicast group on interface usb0.IPv4 with address 192.168.186.1.
syslog:Nov 21 21:10:35 irobot37 avahi-daemon[891]: New relevant interface usb0.IPv4 for mDNS.
syslog:Nov 21 21:10:35 irobot37 avahi-daemon[891]: Registering new address record for fe80::e0d5:92ff:fed4:cf90 on usb0.*.
syslog:Nov 21 21:10:35 irobot37 avahi-daemon[891]: Registering new address record for 192.168.186.1 on usb0.IPv4.
syslog:Nov 21 21:10:36 irobot37 NetworkManager[893]: <info> [1700601036.5290] device (usb0): carrier: link connected
syslog:Nov 21 21:10:36 irobot37 NetworkManager[893]: <info> [1700601036.5302] manager: (usb0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
Binary file cloud-init.log matches
syslog.1:Nov 21 21:10:34 irobot37 systemd-udevd[465]: usb0: Could not generate persistent MAC: No data available
syslog.1:Nov 21 21:10:34 irobot37 systemd-udevd[489]: usb0: Could not generate persistent MAC: No data available
syslog.1:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: IPv6 successfully enabled
syslog.1:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: Link UP
syslog.1:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: Gained carrier
syslog.1:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: Gained IPv6LL
syslog.1:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | e6:06:4d:54:8c:13 |
syslog.1:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | usb0 | True | fe80::e406:4dff:fe54:8c13/64 | . | link | e6:06:4d:54:8c:13 |
syslog.1:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
syslog.1:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
syslog.1:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | 1 | local | :: | usb0 | U |
syslog.1:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | 2 | multicast | :: | usb0 | U |
syslog.1:Nov 21 21:10:35 irobot37 avahi-daemon[891]: Joining mDNS multicast group on interface usb0.IPv6 with address fe80::e406:4dff:fe54:8c13.
syslog.1:Nov 21 21:10:35 irobot37 avahi-daemon[891]: New relevant interface usb0.IPv6 for mDNS.
syslog.1:Nov 21 21:10:35 irobot37 avahi-daemon[891]: Joining mDNS multicast group on interface usb0.IPv4 with address 192.168.186.1.
syslog.1:Nov 21 21:10:35 irobot37 avahi-daemon[891]: New relevant interface usb0.IPv4 for mDNS.
syslog.1:Nov 21 21:10:35 irobot37 avahi-daemon[891]: Registering new address record for fe80::e406:4dff:fe54:8c13 on usb0.*.
syslog.1:Nov 21 21:10:35 irobot37 avahi-daemon[891]: Registering new address record for 192.168.186.1 on usb0.IPv4.
syslog.1:Nov 21 21:10:36 irobot37 NetworkManager[893]: <info> [1700601036.7292] device (usb0): carrier: link connected
syslog.1:Nov 21 21:10:36 irobot37 NetworkManager[893]: <info> [1700601036.7317] manager: (usb0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
syslog.1:Nov 21 21:10:34 irobot37 systemd-udevd[467]: usb0: Could not generate persistent MAC: No data available
syslog.1:Nov 21 21:10:34 irobot37 systemd-udevd[493]: usb0: Could not generate persistent MAC: No data available
syslog.1:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: IPv6 successfully enabled
syslog.1:Nov 21 21:10:34 irobot37 systemd-networkd[856]: usb0: Link UP
syslog.1:Nov 21 21:10:34 irobot37 cloud-init[863]: ci-info: | usb0 | False | . | . | . | 9e:f9:ae:42:f1:e9 |
syslog.1:Nov 21 21:10:36 irobot37 dnsmasq[1031]: warning: interface usb0 does not currently exist
syslog.1:Nov 21 21:10:36 irobot37 NetworkManager[893]: <info> [1700601036.6898] manager: (usb0): new Ethernet device (/org/freedesktop/NetworkManager/Devices/3)
syslog.1:Nov 21 21:10:39 irobot37 systemd-networkd[856]: usb0: Gained carrier
syslog.1:Nov 21 21:10:39 irobot37 NetworkManager[893]: <info> [1700601039.2026] device (usb0): carrier: link connected
syslog.1:Nov 21 21:10:39 irobot37 kernel: [ 23.903330] IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
syslog.1:Nov 21 21:10:39 irobot37 avahi-daemon[891]: Joining mDNS multicast group on interface usb0.IPv4 with address 192.168.186.1.
syslog.1:Nov 21 21:10:39 irobot37 avahi-daemon[891]: New relevant interface usb0.IPv4 for mDNS.
syslog.1:Nov 21 21:10:39 irobot37 avahi-daemon[891]: Registering new address record for 192.168.186.1 on usb0.IPv4.
syslog.1:Nov 21 21:10:40 irobot37 avahi-daemon[891]: Joining mDNS multicast group on interface usb0.IPv6 with address fe80::9cf9:aeff:fe42:f1e9.
syslog.1:Nov 21 21:10:40 irobot37 avahi-daemon[891]: New relevant interface usb0.IPv6 for mDNS.
syslog.1:Nov 21 21:10:40 irobot37 systemd-networkd[856]: usb0: Gained IPv6LL
syslog.1:Nov 21 21:10:40 irobot37 avahi-daemon[891]: Registering new address record for fe80::9cf9:aeff:fe42:f1e9 on usb0.*.
boot.log.2:[ 18.105962] cloud-init[864]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | 86:e5:52:a5:2d:5e |
boot.log.2:[ 18.106137] cloud-init[864]: ci-info: | usb0 | True | fe80::84e5:52ff:fea5:2d5e/64 | . | link | 86:e5:52:a5:2d:5e |
boot.log.2:[ 18.107370] cloud-init[864]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.2:[ 18.108450] cloud-init[864]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.2:[ 18.108630] cloud-init[864]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.2:[ 18.108909] cloud-init[864]: ci-info: | 2 | multicast | :: | usb0 | U |
boot.log.2:[ 18.318767] cloud-init[865]: ci-info: | usb0 | False | . | . | . | 56:bf:61:3a:17:23 |
Binary file cloud-init-output.log matches
dmesg.0:[ 23.903330] kernel: IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
boot.log.1:[ 18.385766] cloud-init[863]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | e6:06:4d:54:8c:13 |
boot.log.1:[ 18.385939] cloud-init[863]: ci-info: | usb0 | True | fe80::e406:4dff:fe54:8c13/64 | . | link | e6:06:4d:54:8c:13 |
boot.log.1:[ 18.387103] cloud-init[863]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.1:[ 18.388222] cloud-init[863]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.1:[ 18.388399] cloud-init[863]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.1:[ 18.388666] cloud-init[863]: ci-info: | 2 | multicast | :: | usb0 | U |
boot.log.1:[ 18.342028] cloud-init[863]: ci-info: | usb0 | False | . | . | . | 9e:f9:ae:42:f1:e9 |
boot.log.4:[ 18.861778] cloud-init[865]: ci-info: | usb0 | False | . | . | . | be:f9:dc:ba:b6:96 |
boot.log.4:[ 17.584960] cloud-init[865]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | 1a:29:01:90:d4:fe |
boot.log.4:[ 17.585129] cloud-init[865]: ci-info: | usb0 | True | fe80::1829:1ff:fe90:d4fe/64 | . | link | 1a:29:01:90:d4:fe |
boot.log.4:[ 17.586338] cloud-init[865]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.4:[ 17.587372] cloud-init[865]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.4:[ 17.587542] cloud-init[865]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.4:[ 17.587728] cloud-init[865]: ci-info: | 2 | multicast | :: | usb0 | U |
boot.log.4:[ 18.573655] cloud-init[863]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | a2:30:a8:3d:9c:0c |
boot.log.4:[ 18.573840] cloud-init[863]: ci-info: | usb0 | True | fe80::a030:a8ff:fe3d:9c0c/64 | . | link | a2:30:a8:3d:9c:0c |
boot.log.4:[ 18.575074] cloud-init[863]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.4:[ 18.576197] cloud-init[863]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.4:[ 18.576372] cloud-init[863]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.4:[ 18.576660] cloud-init[863]: ci-info: | 2 | multicast | :: | usb0 | U |
boot.log.4:[ 18.467098] cloud-init[862]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | 6a:b3:27:c1:68:ff |
boot.log.4:[ 18.467273] cloud-init[862]: ci-info: | usb0 | True | fe80::68b3:27ff:fec1:68ff/64 | . | link | 6a:b3:27:c1:68:ff |
boot.log.4:[ 18.468597] cloud-init[862]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.4:[ 18.469610] cloud-init[862]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.4:[ 18.469808] cloud-init[862]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.4:[ 18.470056] cloud-init[862]: ci-info: | 2 | multicast | :: | usb0 | U |
boot.log.4:[ 18.225372] cloud-init[864]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | fe:b3:01:24:4b:72 |
boot.log.4:[ 18.225548] cloud-init[864]: ci-info: | usb0 | True | fe80::fcb3:1ff:fe24:4b72/64 | . | link | fe:b3:01:24:4b:72 |
boot.log.4:[ 18.226853] cloud-init[864]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.4:[ 18.227914] cloud-init[864]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.4:[ 18.228173] cloud-init[864]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.4:[ 18.228443] cloud-init[864]: ci-info: | 2 | multicast | :: | usb0 | U |
boot.log.4:[ 19.062482] cloud-init[865]: ci-info: | usb0 | False | . | . | . | de:e7:40:2a:bd:f0 |
boot.log.4:[ 18.037827] cloud-init[862]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | 22:23:53:f6:93:a7 |
boot.log.4:[ 18.037996] cloud-init[862]: ci-info: | usb0 | True | fe80::2023:53ff:fef6:93a7/64 | . | link | 22:23:53:f6:93:a7 |
boot.log.4:[ 18.039152] cloud-init[862]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.4:[ 18.040253] cloud-init[862]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.4:[ 18.040438] cloud-init[862]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.4:[ 18.040673] cloud-init[862]: ci-info: | 2 | multicast | :: | usb0 | U |
boot.log.4:[ 18.987299] cloud-init[862]: ci-info: | usb0 | False | . | . | . | 02:1f:08:9e:0c:df |
boot.log.4:[ 17.653933] cloud-init[866]: ci-info: | usb0 | False | . | . | . | be:0a:44:f0:02:3a |
boot.log.4:[ 17.932676] cloud-init[866]: ci-info: | usb0 | False | . | . | . | 36:6b:c9:09:34:eb |
boot.log.4:[ 17.994568] cloud-init[868]: ci-info: | usb0 | False | . | . | . | 56:e0:99:90:ab:7c |
boot.log.7:[ 19.653717] cloud-init[860]: ci-info: | usb0 | False | . | . | . | 3e:c2:97:7d:bf:89 |
boot.log.7:[ 19.249542] cloud-init[860]: ci-info: | usb0 | False | . | . | . | 6e:0f:66:6d:64:b6 |
boot.log.7:[ 17.498905] cloud-init[860]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | 92:2e:05:db:fd:87 |
boot.log.7:[ 17.499082] cloud-init[860]: ci-info: | usb0 | True | fe80::902e:5ff:fedb:fd87/64 | . | link | 92:2e:05:db:fd:87 |
boot.log.7:[ 17.500349] cloud-init[860]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.7:[ 17.501434] cloud-init[860]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.7:[ 17.501601] cloud-init[860]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.7:[ 17.501864] cloud-init[860]: ci-info: | 2 | multicast | :: | usb0 | U |
boot.log.3:[ 18.365397] cloud-init[862]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | 52:78:8a:92:75:64 |
boot.log.3:[ 18.365564] cloud-init[862]: ci-info: | usb0 | True | fe80::5078:8aff:fe92:7564/64 | . | link | 52:78:8a:92:75:64 |
boot.log.3:[ 18.366777] cloud-init[862]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.3:[ 18.367837] cloud-init[862]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.3:[ 18.368049] cloud-init[862]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.3:[ 18.368327] cloud-init[862]: ci-info: | 2 | multicast | :: | usb0 | U |
boot.log.3:[ 18.619399] cloud-init[865]: ci-info: | usb0 | False | . | . | . | 52:b8:24:cb:16:88 |
boot.log.3:[ 18.964443] cloud-init[863]: ci-info: | usb0 | False | . | . | . | 62:93:a3:85:16:e6 |
boot.log.3:[ 18.949899] cloud-init[861]: ci-info: | usb0 | True | 192.168.186.1 | 255.255.255.0 | global | ce:44:bc:3f:94:4c |
boot.log.3:[ 18.950077] cloud-init[861]: ci-info: | usb0 | True | fe80::cc44:bcff:fe3f:944c/64 | . | link | ce:44:bc:3f:94:4c |
boot.log.3:[ 18.951285] cloud-init[861]: ci-info: | 0 | 192.168.186.0 | 0.0.0.0 | 255.255.255.0 | usb0 | U |
boot.log.3:[ 18.952348] cloud-init[861]: ci-info: | 0 | fe80::/64 | :: | usb0 | U |
boot.log.3:[ 18.952557] cloud-init[861]: ci-info: | 1 | local | :: | usb0 | U |
boot.log.3:[ 18.952817] cloud-init[861]: ci-info: | 2 | multicast | :: | usb0 | U |
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007dbf7d4-80576c1b96e656c1.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007db47b2-66fd410d43ad2e44.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@bba802095fec4106a3a9501b6c37ec9e-0000000000000001-00060ab007d88618.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007da1c89-f89692f3146c1eb0.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007da2f32-14b129b09c1176c7.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007dcb45e-bdf274db77fe1b71.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@918f695977bb42dea563209fa2c153a7-0000000000000001-00060ab007d80152.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007da9ac5-bc5bda638157ed88.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@32ea739536d041dc83e63bf850a4d206-0000000000000001-00060ab007d98ca2.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@451d51780ee24f21b5820b2401460797-0000000000000001-00060ab007d87c3f.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@bac5ca2d18e34a45b7e5a95f206d66c3-0000000000000001-00060ab007d6c770.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007daa4c8-314281595745781b.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@14a81d54bc6d4778971c99e02e1a8588-0000000000000001-00060ab007d7baa8.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@a4556b3a22d84d4eafad4680a9793838-0000000000000001-00060ab007d616d5.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007da156e-3268e6cd54425a76.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@dc17dc2fd22449e4838aa3949c26d22d-0000000000000001-00060ab007d93f41.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007daf346-badd8a40d5544f22.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@cd1c167f05fa4a73aabf3e8468b38bed-0000000000000001-00060ab007dbd726.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007db78a8-a7b5d832709a910f.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007d98cd4-2db565dcfac26dd0.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@6a0eb3d1277e45e182248c6ece4436b2-0000000000000001-00060ab007d8385a.journal matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007dad465-777f580489585717.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@00060ab007da0e51-729199596f83fd3c.journal~ matches
Binary file journal/49189fa4294a452084601077333f5a81/system@d5a38c92ae824207bd5c58c9c795ec45-0000000000000001-00060ab007da3e43.journal matches
auth.log:Apr 11 09:27:04 irobot37 sudo: ubuntu : TTY=pts/0 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/grep -r usb0
auth.log:Apr 15 14:17:50 irobot37 sudo: ubuntu : TTY=pts/1 ; PWD=/var/log ; USER=root ; COMMAND=/usr/bin/grep -r usb0
boot.log.6:[ 17.468695] cloud-init[860]: ci-info: | usb0 | False | . | . | . | 06:37:e3:aa:a4:19 |
kern.log.1:Nov 21 21:10:38 irobot37 kernel: [ 23.933870] IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
kern.log.1:Nov 21 21:10:38 irobot37 kernel: [ 24.029286] IPv6: ADDRCONF(NETDEV_CHANGE): usb0: link becomes ready
My experience has generally been that any restart of the Pi can disrupt the ROS 2 connection with the Create 3. I think it's a weird middleware/ROS 2 issue. Whenever I restart the Pi, I also reboot the Create 3 after doing so.
Hi @brianabouchard, thanks have just found the same, restarting the Pi reliably generates the issue.
Can the network stack on the robot be automatically restarted if it looses usb0 comms, it's always the same error (tev: ddsi_udp_conn_write to udp/239.255.0.1:9650 failed with retcode -1)? As the boot cycle of the robot is very long and its quite easy to forget to do this...
I can't speak for the iRobot team and whether or not that would be possible. But as a workaround, it should be possible to setup a service that runs on boot and executes either curl -X POST http://192.168.186.2/api/reboot
or curl -X POST http://192.168.186.2/api/restart-app
. I haven't tested the latter for this issue, but in theory, restarting the application rather than fully rebooting the robot may resolve the issue and would be faster.
thanks @brianabouchard will give that a go
Hi @lukeopteran, the reason you don't see the topics all of the time is due to discovery issues and the inability of the Create 3 to parse these in time in a DDS network.
We have made a video explaining why this issue happens, and how it can be solved by simply pulling and running a docker container.
Although it's made for the TurtleBot 4, it applies to any Create 3 with a computing unit attached.
The basic idea is separating the ROS_DOMAIN_ID
of the RPi4 and the Create 3, and connecting them through two zenoh bridges in order to isolate and protect the Create 3.
I hope it helps!
https://www.youtube.com/watch?v=xmK2I0D5sas
Check out our docker image here
Raspberry Pi configuration (use turtlebot4-setup
tool):
RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
ROS_DOMAIN_ID=0
CYCLONEDDS_URI=[]
(Empty).Create 3 configuration (Access web server in a browser: 192.168.1.XX:8080
):
RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
ROS_DOMAIN_ID=1
Use ROS Discovery Server
.docker-compose.yaml
✅️ The best way to run this image is with the following docker-compose.yaml
:
services:
zenoh-bridge-turtlebot4:
image: theconstructai/zenoh-bridge-ros2dds:turtlebot4
container_name: zenoh-bridge-turtlebot4
network_mode: "host"
restart: always # Ensures the container restarts on reboot
docker compose -f /path/to/docker-compose.yaml up -d
.ros2 topic list
.
We frequently have to reboot the robot (best via the UI) to discover the ROS2 topics, they are visible initially but then disappear after an hour or two.
I don't know if it is related to this message in the log files: Nov 14 13:52:03 iRobot-A8BDD22042AB4DCBBA85072EAED55E2E user.notice create-platform: 1699969923.059798 [9] tev: ddsi_udp_conn_write to udp/239.255.0.1:9650 failed with retcode -1
Robot setup
Firmware: G5.2 Host: Pi4 inside Ubuntu 20.04 on usb0 ROS galactic, running chrony as a ntp server