[10885 ] ros2 foxy discovery server over tcp transport

yuraSomatic commented 3 years ago

I'm trying to setup ros2 foxy discovery server with tcp transport, using xml files. It's wortking via udpv4, but not working via tcpv4.

Current, Expected Behavior

I'm using this xml for server setup and running server via ros2 daemon start:

And using this xml for clients setup

When I'm changing metatrafficUnicastLocatorList in both files from udpv4 to tcpv4 - ros2 topic list command stop detecting my clients topics. What i suppose to do to make it work? In all tutorials you always using DS + udpv4, but not tcpv4, please help :|

yuraSomatic commented 3 years ago

maybe someone has any little advices? Im tired fighting this issue 2 weeks with no proggress (

MiguelBarro commented 3 years ago

Using tcpv4 requires creating a specific transport to handle them. Please read this example xml as guidance.

yuraSomatic commented 3 years ago

Thank's for response !!!

the example xml proposals

1) I can't find this link from official tutorials, i think it's may be helpfull for other peoples too 2) There is no client xml configuration, just c++ code 3) Some bugs in that example code port numbers, for example in TCP transport code setup for a client there is IPLocator::setLogicalPort(server_address, 64863); but below in explanation

Note that here the server_address locator specifies 65215 as a logical port

Finally i make it work in LAN, using xml configuration below

server

<?xml version="1.0" encoding="UTF-8" ?>

<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
        <transport_descriptors>
            <transport_descriptor>
                <transport_id>TCPv4_SERVER</transport_id>
                <type>TCPv4</type>
                <listening_ports>
                    <port>9843</port>
                </listening_ports>
            </transport_descriptor>
        </transport_descriptors>

        <participant profile_name="participant_server" is_default_profile="true">
            <rtps>
                <prefix>
                    4D.49.47.55.45.4c.5f.42.41.53.52.4f <!-- 12-byte server unique id -->
                </prefix>
                <userTransports>
                    <transport_id>TCPv4_SERVER</transport_id>
                </userTransports>
                <useBuiltinTransports>false</useBuiltinTransports>

                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SERVER</discoveryProtocol> <!-- participant is metatrafic hub -->
                        <leaseDuration>
                            <sec>DURATION_INFINITY</sec>
                        </leaseDuration>
                    </discovery_config>

                    <metatrafficUnicastLocatorList> <!-- here server listening for clients metatrafic -->
                        <locator>
                            <tcpv4>
                                <!-- if no address is provided the server would export all its public interfaces as address -->
                                <!-- this is a logical port, the physical one is specify as listening port above -->
                                <port>64863</port>
                            </tcpv4>
                        </locator>
                    </metatrafficUnicastLocatorList>
                </builtin>
            </rtps>
        </participant>
    </profiles>
</dds>

client

<?xml version="1.0" encoding="UTF-8" ?>
<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
        <transport_descriptors>
            <transport_descriptor>
                <transport_id>TCPv4_CLI</transport_id>
                <type>TCPv4</type>
                <listening_ports>
                    <port>49153</port>
                </listening_ports>
            </transport_descriptor>
        </transport_descriptors>

        <participant profile_name="participant_client"  is_default_profile="true">
            <rtps>
                <userTransports>
                    <transport_id>TCPv4_CLI</transport_id>
                </userTransports>
                <useBuiltinTransports>false</useBuiltinTransports>
                <builtin>
                    <discovery_config>

                        <discoveryProtocol>CLIENT</discoveryProtocol>
                        <leaseDuration>
                            <sec>DURATION_INFINITY</sec>
                        </leaseDuration>

                        <discoveryServersList>
                            <RemoteServer prefix="4D.49.47.55.45.4c.5f.42.41.53.52.4f"> <!-- must match server's id -->
                                <metatrafficUnicastLocatorList> 
                                    <locator>
                                        <tcpv4> <!-- must match server's (ip, port) -->
                                            <address>192.168.8.100</address>
                                            <port>64863</port>
                                            <physical_port>9843</physical_port>
                                        </tcpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                        </discoveryServersList>

                    </discovery_config>
                </builtin>
            </rtps>
        </participant>
    </profiles>
</dds>

issue current status

But I still can't make it work in WAN. I just replaced <address>192.168.8.100</address> with my server public address and enabled connection via ubuntu firewall. Should I use different transport descriptor or domain participant configuration for this like <wan_addr> or <wan_address> or other dds settings to make it work?

yuraSomatic commented 3 years ago

When I just specifying <wan_addr> in discovery server <transport_descriptor> xml tag it fails with following log:

yuraSomatic commented 3 years ago

sudo lsof -i -P -n output looks like:

WAN

client

talker 33884 uri 10u IPv4 407477 0t0 TCP *:49154 (LISTEN) talker 33884 uri 11u IPv4 413203 0t0 TCP 192.168.8.100:36216->SERVER_PUBLIC_IP:9843 (ESTABLISHED)

listener 33860 uri 10u IPv4 415127 0t0 TCP *:49153 (LISTEN) listener 33860 uri 11u IPv4 415527 0t0 TCP 192.168.8.100:36214->SERVER_PUBLIC_IP:9843 (ESTABLISHED)

server

discovery 2021 jetson 12u IPv4 174097 0t0 TCP *:9843 (LISTEN) discovery 2021 jetson 13u IPv4 170555 0t0 TCP 192.168.8.100:9843->CLIENT_PUBLIC_IP:36214 (ESTABLISHED) discovery 2021 jetson 14u IPv4 170556 0t0 TCP 192.168.8.100:9843->CLIENT_PUBLIC_IP:36216 (ESTABLISHED)

LOCAL (1 host)

client

talker 36307 uri 10u IPv4 459717 0t0 TCP *:49154 (LISTEN) talker 36307 uri 11u IPv4 459728 0t0 TCP 127.0.0.1:60630->127.0.0.1:9843 (ESTABLISHED) talker 36307 uri 12u IPv4 468618 0t0 TCP 127.0.0.1:55276->127.0.0.1:49153 (ESTABLISHED) talker 36307 uri 13u IPv4 459760 0t0 TCP 127.0.0.1:49154->127.0.0.1:40954 (ESTABLISHED)

listener 36323 uri 10u IPv4 459748 0t0 TCP *:49153 (LISTEN) listener 36323 uri 11u IPv4 459759 0t0 TCP 127.0.0.1:60632->127.0.0.1:9843 (ESTABLISHED) listener 36323 uri 12u IPv4 469283 0t0 TCP 127.0.0.1:40954->127.0.0.1:49154 (ESTABLISHED) listener 36323 uri 13u IPv4 466569 0t0 TCP 127.0.0.1:49153->127.0.0.1:55276 (ESTABLISHED)

server

discovery 36285 uri 10u IPv4 466332 0t0 TCP *:9843 (LISTEN) discovery 36285 uri 11u IPv4 469279 0t0 TCP 127.0.0.1:9843->127.0.0.1:60630 (ESTABLISHED) discovery 36285 uri 12u IPv4 468615 0t0 TCP 127.0.0.1:9843->127.0.0.1:60632 (ESTABLISHED)

MiguelBarro commented 3 years ago

In order to clarify the WAN setup we are going to provide a specific example (next week we'll update the docs):

WAN network with mask 192.168.36.X. Here:
- Subscriber client with address 192.168.36.54
- LAN router with address 192.168.36.53
LAN network with mask 192.168.42.X. Here the server and a publisher in the same address 192.168.42.73.

WAN subscriber client profile

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    <transport_descriptors>
      <transport_descriptor>
          <transport_id>WLAN tcp transport</transport_id>
          <type>TCPv4</type>
          <listening_ports>
              <port>64863</port> <!-- subscriber devoted tcp port -->
          </listening_ports>
          <wan_addr>192.168.36.54</wan_addr> <!-- This subscriber computer WAN address -->
      </transport_descriptor>
    </transport_descriptors>

    <participant profile_name="WLAN client" >
      <rtps>
        <userTransports>
              <transport_id>WLAN tcp transport</transport_id>
        </userTransports>
        <useBuiltinTransports>false</useBuiltinTransports>
        <builtin>
            <discovery_config>
                <discoveryProtocol>CLIENT</discoveryProtocol>
                <discoveryServersList>
                    <RemoteServer prefix="4D.49.47.55.45.4c.5f.42.41.52.52.4f">
                        <metatrafficUnicastLocatorList>
                            <locator>
                                <tcpv4>
                                    <address>192.168.36.53</address>
                                    <port>65215</port>
                                    <physical_port>5100</physical_port>
                                </tcpv4>
                            </locator>
                        </metatrafficUnicastLocatorList>
                    </RemoteServer>
                </discoveryServersList>
            </discovery_config>
        </builtin>
      </rtps>
    </participant>
</profiles>

LAN server and publisher client profiles

<?xml version="1.0" encoding="UTF-8" ?>
<profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
    <transport_descriptors>

      <transport_descriptor>
          <transport_id>LAN server tcp transport</transport_id>
          <type>TCPv4</type>
          <listening_ports>
              <port>5100</port> <!-- server devoted tcp port -->
          </listening_ports>
          <wan_addr>192.168.36.53</wan_addr> <!-- server computer WAN address -->
      </transport_descriptor>

      <transport_descriptor>
          <transport_id>LAN publisher tcp transport</transport_id>
          <type>TCPv4</type>
          <listening_ports>
              <port>64752</port> <!-- publisher devoted tcp port -->
          </listening_ports>
          <wan_addr>192.168.36.53</wan_addr> <!-- publisher computer WAN address -->
      </transport_descriptor>

    </transport_descriptors>

    <participant profile_name="LAN client" >
      <rtps>
          <userTransports>
              <transport_id>LAN publisher tcp transport</transport_id>
          </userTransports>
          <useBuiltinTransports>false</useBuiltinTransports>
          <builtin>
              <discovery_config>
                  <discoveryProtocol>CLIENT</discoveryProtocol>
                  <discoveryServersList>
                      <RemoteServer prefix="4D.49.47.55.45.4c.5f.42.41.52.52.4f">
                          <metatrafficUnicastLocatorList>
                              <locator>
                                  <tcpv4>
                                      <address>192.168.42.73</address>
                                      <port>65215</port>
                                      <physical_port>5100</physical_port>
                                  </tcpv4>
                              </locator>
                          </metatrafficUnicastLocatorList>
                      </RemoteServer>
                  </discoveryServersList>
              </discovery_config>
          </builtin>
      </rtps>
    </participant>

    <participant profile_name="LAN server">
        <rtps>
            <prefix>
                4D.49.47.55.45.4c.5f.42.41.52.52.4f
            </prefix>
            <userTransports>
                  <transport_id>LAN server tcp transport</transport_id>
            </userTransports>
            <useBuiltinTransports>false</useBuiltinTransports>
            <builtin>
                <discovery_config>
                    <discoveryProtocol>SERVER</discoveryProtocol>
                </discovery_config>
                <metatrafficUnicastLocatorList>
                    <locator>
                        <tcpv4>
                            <address>192.168.42.73</address>
                            <port>65215</port>
                            <physical_port>5100</physical_port>
                            <wan_address>192.168.36.53</wan_address>
                        </tcpv4>
                    </locator>
                </metatrafficUnicastLocatorList>
            </builtin>
        </rtps>
    </participant>
</profiles>

Some Auxiliary tools

Using tcp across a NAT requires specific firewall and router set up (port forwarding). In order to check it we advice to use:

ShapesDemo which provides a dialog base GUI. This doesn't use the discovery server thus, it is easy to detect specific tcp transport issues.
HelloWorldExampleDS whose cli has been extended to test discovery server over WAN. No xml config files are required which simplifies detecting hardware and OS set up issues.

yuraSomatic commented 3 years ago

That works, thanks! I also added is_default_profile property to discovery server and wan client participant tags, to make it work with ros foxy: <participant profile_name="LAN server" is_default_profile="true"> <participant profile_name="participant_client" is_default_profile="true">

EduPonz commented 3 years ago

Hi @yuraSomatic ,

On a side note, we do not recommend running the Discovery Server on the daemon node, since it shuts down after 2 hours of inactivity. You may want either a dedicated node, or one of your own nodes as the server

GilmarCorreia commented 1 year ago

Hello ros-developers. I'm trying to run the same xml available above for LAN, but I getting the same error over and over. Which steps should I follow?

machine 1 (192.168.x.y) - terminal 1

using this server file:

<?xml version="1.0" encoding="UTF-8" ?>

<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
        <transport_descriptors>
            <transport_descriptor>
                <transport_id>TCPv4_SERVER</transport_id>
                <type>TCPv4</type>
                <listening_ports>
                    <port>9843</port>
                </listening_ports>
            </transport_descriptor>
        </transport_descriptors>

        <participant profile_name="participant_server" is_default_profile="true">
            <rtps>
                <prefix>
                    4D.49.47.55.45.4c.5f.42.41.53.52.4f <!-- 12-byte server unique id -->
                </prefix>
                <userTransports>
                    <transport_id>TCPv4_SERVER</transport_id>
                </userTransports>
                <useBuiltinTransports>false</useBuiltinTransports>

                <builtin>
                    <discovery_config>
                        <discoveryProtocol>SERVER</discoveryProtocol> <!-- participant is metatrafic hub -->
                        <leaseDuration>
                            <sec>DURATION_INFINITY</sec>
                        </leaseDuration>
                    </discovery_config>

                    <metatrafficUnicastLocatorList> <!-- here server listening for clients metatrafic -->
                        <locator>
                            <tcpv4>
                                <!-- if no address is provided the server would export all its public interfaces as address -->
                                <!-- this is a logical port, the physical one is specify as listening port above -->
                                                                <address>0.0.0.0</address>
                                <port>64863</port>
                            </tcpv4>
                        </locator>
                    </metatrafficUnicastLocatorList>
                </builtin>
            </rtps>
        </participant>
    </profiles>
</dds>

source ros_distribtion/setup.bash
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
export FASTRTPS_DEFAULT_PROFILES_FILE=fastdds_server.xml
ros2 daemon stop
ros2 daemon start
ros2 run demo_nodes_cpp listener

2022-09-20 16:57:36.446 [RTCP_MSG_OUT Error] TCPTransport Error binding at port: (9843) with msg: bind: Address already in use -> Function create_acceptor_socket

In the client machine:

machine 2 (192.168.w.z) - terminal 1

<?xml version="1.0" encoding="UTF-8" ?>
<dds>
    <profiles xmlns="http://www.eprosima.com/XMLSchemas/fastRTPS_Profiles">
        <transport_descriptors>
            <transport_descriptor>
                <transport_id>TCPv4_CLI</transport_id>
                <type>TCPv4</type>
                <listening_ports>
                    <port>49153</port>
                </listening_ports>
            </transport_descriptor>
        </transport_descriptors>

        <participant profile_name="participant_client"  is_default_profile="true">
            <rtps>
                <userTransports>
                    <transport_id>TCPv4_CLI</transport_id>
                </userTransports>
                <useBuiltinTransports>false</useBuiltinTransports>
                <builtin>
                    <discovery_config>

                        <discoveryProtocol>CLIENT</discoveryProtocol>
                        <leaseDuration>
                            <sec>DURATION_INFINITY</sec>
                        </leaseDuration>

                        <discoveryServersList>
                            <RemoteServer prefix="4D.49.47.55.45.4c.5f.42.41.53.52.4f"> <!-- must match server's id -->
                                <metatrafficUnicastLocatorList> 
                                    <locator>
                                        <tcpv4> <!-- must match server's (ip, port) -->
                                            <address>192.168.x.y</address>
                                            <port>64863</port>
                                            <physical_port>9843</physical_port>
                                        </tcpv4>
                                    </locator>
                                </metatrafficUnicastLocatorList>
                            </RemoteServer>
                        </discoveryServersList>

                    </discovery_config>
                </builtin>
            </rtps>
        </participant>
    </profiles>
</dds>

source ros_distribtion/setup.bash
export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
export FASTRTPS_DEFAULT_PROFILES_FILE=fastdds_client.xml
ros2 daemon stop
ros2 daemon start
ros2 run demo_nodes_cpp talker

2022-09-20 16:57:36.446 [RTCP_MSG_OUT Error] TCPTransport Error binding at port: (49153) with msg: bind: Address already in use -> Function create_acceptor_socket

Both machines cannot communicate itself.

EduPonz commented 1 year ago

Hi @GilmarCorreia ,

It seems that some other process has that port already taken. Under Ubuntu, you may use netstat to check the process that is occupying the TCP port.

magladko commented 9 months ago

Anyone here tried tunneling the TCP traffic through SSH? I try different options for ssh port forwarding, but can't make it work. Would it be at all possible?

eProsima / Fast-DDS