Zenoh unable to connect over loopback interface in Galactic Cyclone DDS (but able to on Foxy)

sisaha9 commented 2 years ago

Bug report

I am running ROS2 on 2 hosts. Both hosts are using Cyclone DDS with

Operating System:
- Ubuntu 20.04 with ROS2 Galactic
Installation type:
- Binary
Version or commit hash:
- Not sure but the latest binary
DDS implementation:
- Cyclone DDS
Client library (if applicable):
- rclcpp

Steps to reproduce issue

Using the config detailed here

<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" 
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
            xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
    <Domain id="any">
        <General>
            <NetworkInterfaceAddress>lo</NetworkInterfaceAddress>  <!-- This line -->
            <AllowMulticast>default</AllowMulticast>
            <MaxMessageSize>65500B</MaxMessageSize>
            <FragmentSize>4000B</FragmentSize>
        </General>
        <Internal>
            <Watermarks>
                <WhcHigh>500kB</WhcHigh>
            </Watermarks>
            <AssumeMulticastCapable>lo</AssumeMulticastCapable>    <!-- And this line -->
        </Internal>
    </Domain>
</CycloneDDS>

The other config (in the basestation) is the following

<?xml version="1.0" encoding="UTF-8" ?>
<CycloneDDS xmlns="https://cdds.io/config" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/master/etc/cyclonedds.xsd">
    <Domain id="any">
        <General>
            <NetworkInterfaceAddress>auto</NetworkInterfaceAddress>
            <AllowMulticast>default</AllowMulticast>
            <MaxMessageSize>65500B</MaxMessageSize>
            <FragmentSize>4000B</FragmentSize>
        </General>
        <Internal>
            <Watermarks>
                <WhcHigh>500kB</WhcHigh>
            </Watermarks>
        </Internal>
        <Tracing>
            <Verbosity>config</Verbosity>
            <OutputFile>stdout</OutputFile>
        </Tracing>
    </Domain>
</CycloneDDS>

Then run Zenoh on each side (using the IAC2 branch) For the racecar

ROS_DOMAIN_ID=71 RUST_LOG=debug zenoh-bridge-dds -d 71 -m peer --no-multicast-scouting -l udp/0.0.0.0:7447

For the basestation

ROS_DOMAIN_ID=72 RUST_LOG=debug zenoh-bridge-dds -d 72 -m peer --no-multicast-scouting -l udp/0.0.0.0:7447 -e udp/10.42.7.200:7447

where 10.42.7.200 is the IP of the racecar

Expected behavior

It works as mentioned in the blog post. We can connect using Zenoh and share messages b/w the basestation and the racecar

Actual behavior

It does not work like in the blog post. I am unable to connect via Zenoh and can't share messages

Additional information

It works in Foxy. It works when using Network interface auto in both the racecar and basestation. It works when using another DDS

joespeed commented 2 years ago

@JEnoch @eboasson @allenh1 help please to get Indy Autonomous Challenge university racing teams get their ROS 2 Galactic Cyclone DDS + Zenoh config working. @JWhitleyWork set it up in the racecars with Foxy a few months ago. TUM moved to Galactic w/o Zenoh. Now the University of Hawaii + UC San Diego and other teams are moving to Galactic with Zenoh.

JEnoch commented 2 years ago

@sisaha9 Can you please provide the logs of the 2 zenoh-bridge-dds.

sisaha9 commented 2 years ago

Basestation keeps outputting this:

[2021-12-08T16:13:38Z ERROR zenoh::net::routing::pubsub] Route data with unknown rid 53!
[2021-12-08T16:13:39Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 8AD4A58242854751A903A8406D26B71E
[2021-12-08T16:13:40Z ERROR zenoh::net::routing::pubsub] Route data with unknown rid 53!
[2021-12-08T16:13:41Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 8AD4A58242854751A903A8406D26B71E
[2021-12-08T16:13:42Z ERROR zenoh::net::routing::pubsub] Route data with unknown rid 53!
[2021-12-08T16:13:43Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 8AD4A58242854751A903A8406D26B71E
[2021-12-08T16:13:44Z ERROR zenoh::net::routing::pubsub] Route data with unknown rid 53!
[2021-12-08T16:13:46Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 8AD4A58242854751A903A8406D26B71E
[2021-12-08T16:13:47Z ERROR zenoh::net::routing::pubsub] Route data with unknown rid 53!
[2021-12-08T16:13:48Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 8AD4A58242854751A903A8406D26B71E
[2021-12-08T16:13:49Z ERROR zenoh::net::routing::pubsub] Route data with unknown rid 53!
[2021-12-08T16:13:50Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 8AD4A58242854751A903A8406D26B71E
[2021-12-08T16:13:51Z ERROR zenoh::net::routing::pubsub] Route data with unknown rid 53!
[2021-12-08T16:13:52Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 8AD4A58242854751A903A8406D26B71E
[2021-12-08T16:13:53Z ERROR zenoh::net::routing::pubsub] Route data with unknown rid 53!

Racecar outputs this:

Dec 08 08:14:26 art-car07 bash[2172]: [2021-12-08T16:14:26Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:29 art-car07 bash[2172]: [2021-12-08T16:14:29Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:31 art-car07 bash[2172]: [2021-12-08T16:14:31Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:33 art-car07 bash[2172]: [2021-12-08T16:14:33Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:35 art-car07 bash[2172]: [2021-12-08T16:14:35Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:38 art-car07 bash[2172]: [2021-12-08T16:14:38Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:40 art-car07 bash[2172]: [2021-12-08T16:14:40Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:42 art-car07 bash[2172]: [2021-12-08T16:14:42Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:44 art-car07 bash[2172]: [2021-12-08T16:14:44Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:47 art-car07 bash[2172]: [2021-12-08T16:14:47Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:49 art-car07 bash[2172]: [2021-12-08T16:14:49Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:51 art-car07 bash[2172]: [2021-12-08T16:14:51Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:53 art-car07 bash[2172]: [2021-12-08T16:14:53Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:56 art-car07 bash[2172]: [2021-12-08T16:14:56Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:14:58 art-car07 bash[2172]: [2021-12-08T16:14:58Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:00 art-car07 bash[2172]: [2021-12-08T16:15:00Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:02 art-car07 bash[2172]: [2021-12-08T16:15:02Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:05 art-car07 bash[2172]: [2021-12-08T16:15:05Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:07 art-car07 bash[2172]: [2021-12-08T16:15:07Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:09 art-car07 bash[2172]: [2021-12-08T16:15:09Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:11 art-car07 bash[2172]: [2021-12-08T16:15:11Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:14 art-car07 bash[2172]: [2021-12-08T16:15:14Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:16 art-car07 bash[2172]: [2021-12-08T16:15:16Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:18 art-car07 bash[2172]: [2021-12-08T16:15:18Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:20 art-car07 bash[2172]: [2021-12-08T16:15:20Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:23 art-car07 bash[2172]: [2021-12-08T16:15:23Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:25 art-car07 bash[2172]: [2021-12-08T16:15:25Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B
Dec 08 08:15:27 art-car07 bash[2172]: [2021-12-08T16:15:27Z DEBUG zenoh_ext::net::group] Sending Keep Alive for: 1BA917FEE9B740ABAE2CD3EC8163B87B

JEnoch commented 2 years ago

Actually what would be interesting are the full logs, since at the beginning they show what's discovered or not, and if bridges have established a session with each other. You can send them via email, in reply to @joespeed introduction email.

roncapat commented 1 year ago

Same problem on Humble, there are also deprecation warnings about XML file NetworkInterfaceAddress and AssumeMulticastCapable tags (CycloneDDS 0.11.0).

sisaha9 commented 1 year ago

I should have closed this. Was recently able to fix it with this: https://github.com/eclipse-zenoh/zenoh-plugin-dds/issues/127

ros2 / rmw_cyclonedds