Cannot get messages when communicating crossing different machines and docker containers via multicast by defining Peers

[ipa-rwu]

Bug report

Required Info:

• Operating System:
  • Ubuntu 22.04.3 amd64
• Installation type:
  • binaries
• Version or commit hash:
  • 1.3.4
• DDS implementation:
  • CycloneDDS
• Client library (if applicable):
  • rmw_cyclonedds_cpp

Steps to reproduce issue

My setup as shown below:

You can reproduce it from here

• Run docker compose file

  version: "3"
  networks:
    ros-network:
      external: false
      driver: "bridge"
      ipam:
        config:
          - subnet: "172.18.0.0/24"
            ip_range: "172.18.0.64/26"
            gateway: "172.18.0.1"
  
  services:
  
    talker:
      image: communication_test_nodes_humble:user
      build:
        context: .
        dockerfile: Dockerfile_USER
        args:
          DOCKER_USER: ${USER}
          BASE_IMAGE: ghcr.io/ipa-rwu/communication_test_nodes_humble:latest
      networks:
        ros-network:
          priority: 999
          ipv4_address: 172.18.0.3
      ports:
        - "7312:7412/udp"
        - "7313:7413/udp"
      environment:
        - NETINTERFACE=eth0
        - ROS_DOMAIN_ID=0
        - ROS_SECURITY_ENABLE=false
        - RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
        - CYCLONEDDS_URI=/cyclonedds.xml
      user: 1000:1000
      volumes:
        - ./cyclonedds_talker.xml:/cyclonedds.xml
        - ./talker.log:/home/${USER}/talker.log
      stdin_open: true
      tty: true
      command:
        - bash
        - -c
        - |
          ros2 run demo_nodes_cpp talker
  
    listener:
      image: communication_test_nodes_humble:user
      build:
        context: .
        dockerfile: Dockerfile_USER
        args:
          DOCKER_USER: ${USER}
          BASE_IMAGE: ghcr.io/ipa-rwu/communication_test_nodes_humble:latest
      networks:
        ros-network:
          priority: 999
          ipv4_address: 172.18.0.4
      ports:
        - "7314:7414/udp"
        - "7315:7415/udp"
      environment:
        - NETINTERFACE=eth0
        - ROS_DOMAIN_ID=0
        - ROS_SECURITY_ENABLE=false
        - RMW_IMPLEMENTATION=rmw_cyclonedds_cpp
        - CYCLONEDDS_URI=/cyclonedds.xml
      user: 1000:1000
      volumes:
        - ./cyclonedds_listener.xml:/cyclonedds.xml
        - ./listener.log:/home/${USER}/listener.log
      stdin_open: true
      tty: true
      command:
        - bash
        - -c
        - |
          ros2 run demo_nodes_cpp listener
  

  • Talker in docker container use cyclonedds_talker.xml

    <?xml version="1.0" encoding="UTF-8" ?>
    <CycloneDDS xmlns="https://cdds.io/config"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/iceoryx/etc/cyclonedds.xsd">
    
      <Domain id="any">
          <!-- default setting -->
          <General>
              <Interfaces>
                  <NetworkInterface name="${NETINTERFACE}"/>
              </Interfaces>
              <AllowMulticast>false</AllowMulticast>
              <Transport>udp</Transport>
          </General>
          <Discovery>
              <Peers>
                  <Peer address="talker"/>
                  <Peer address="listener"/>
                  <Peer address="192.168.0.213"/>
              </Peers>
              <ParticipantIndex>1</ParticipantIndex>
          </Discovery>
          <Tracing>
              <Verbosity>fine</Verbosity>
              <OutputFile>talker.log</OutputFile>
          </Tracing>
      </Domain>
    </CycloneDDS>
    

  • Listener in docker container use cyclonedds_listener.xml

    
    <?xml version="1.0" encoding="UTF-8" ?>
    <CycloneDDS xmlns="https://cdds.io/config"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/iceoryx/etc/cyclonedds.xsd">
    
    <Domain id="any">
        <!-- default setting -->
        <General>
            <Interfaces>
                <NetworkInterface name="${NETINTERFACE}"/>
            </Interfaces>
            <AllowMulticast>false</AllowMulticast>
            <Transport>udp</Transport>
        </General>
        <Discovery>
            <Peers>
                <Peer address="talker"/>
                <Peer address="listener"/>
                <Peer address="192.168.0.213"/>
            </Peers>
            <ParticipantIndex>2</ParticipantIndex>
        </Discovery>
        <Tracing>
            <Verbosity>fine</Verbosity>
            <OutputFile>listener.log</OutputFile>
        </Tracing>
    </Domain>
    </CycloneDDS>

• Other laptop usecyclonedds_machine2.xml

  <?xml version="1.0" encoding="UTF-8" ?>
  <CycloneDDS xmlns="https://cdds.io/config"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="https://cdds.io/config https://raw.githubusercontent.com/eclipse-cyclonedds/cyclonedds/iceoryx/etc/cyclonedds.xsd">
  
    <Domain id="any">
        <!-- default setting -->
        <General>
            <Interfaces>
                <NetworkInterface name="wlan0"/>
            </Interfaces>
            <AllowMulticast>false</AllowMulticast>
            <Transport>udp</Transport>
        </General>
        <Discovery>
            <Peers>
                <Peer address="192.168.0.108:7314"/>
                <Peer address="192.168.0.108:7312"/>
                <Peer address="192.168.0.213"/>
            </Peers>
            <ParticipantIndex>1</ParticipantIndex>
        </Discovery>
        <Tracing>
            <Verbosity>fine</Verbosity>
            <OutputFile>machine2.log</OutputFile>
        </Tracing>
    </Domain>
  </CycloneDDS>
  

  • Run command ros2 run demo_nodes_cpp talker

    Expected behavior

    The listener running docker container should get messages from machine2_talker running on the other laptop

Actual behavior

In the log from machine2_talker running on the other laptop, it seems to find the route to communicate with nodes running in containers but got this error

1700260008.210406 [0] dq.builtin: ddsi_rebuild_writer_addrset(110af39:53bfa8dc:f76ce963:301c3): udp/172.18.0.3:7412@3 udp/172.18.0.4:7414@3 (burst size 4294901760 rexmit 283990)
1700260008.210428 [0] dq.builtin: directed SPDP packet -> not responding

In the log from listener running in a container, it wanted to answer

1700260008.217474 [0] dq.builtin: ddsi_rebuild_writer_addrset(110af78:b971ef08:4d53c42b:301c3): udp/172.18.0.3:7412@549 udp/192.168.0.213:7412@549 (burst size 4294901760 rexmit 283990)
1700260008.217612 [0] dq.builtin: broadcasted SPDP packet -> answering

You can full logs: listener_labeled.log, talker_labeled.log, machine2.log

Additional information

When I run netstat -tuln in the container where listener runs, I see there are also some other ports are open such as 54053, 42153, 46557. Are they used by DDS?

$ netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.11:46557        0.0.0.0:*               LISTEN     
udp        0      0 172.18.0.4:54053        0.0.0.0:*                          
udp        0      0 0.0.0.0:7414            0.0.0.0:*                          
udp        0      0 0.0.0.0:7415            0.0.0.0:*                          
udp        0      0 127.0.0.11:42153        0.0.0.0:* 

The result from the container where talker runs

netstat -tuln
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0 127.0.0.11:37417        0.0.0.0:*               LISTEN     
udp        0      0 172.18.0.3:42856        0.0.0.0:*                          
udp        0      0 0.0.0.0:7412            0.0.0.0:*                          
udp        0      0 0.0.0.0:7413            0.0.0.0:*                          
udp        0      0 127.0.0.11:33721        0.0.0.0:*   

Thank you for your help!

[hjrnunes]

@ipa-rwu did you manage to get it working? I'm trying to get an isolated docker container to talk but no luck.

It would be nice to know which ports we need to bind.