Unable to communicate only in certain conditions OPENVPN

[FiIipe]

Hello,

I am trying to setup the my computers to use FastRTPS but I'm having some problems. My setup is the following:

Machine A: Ubuntu 20.04, running ros2 foxy and rmw_fastrtps_cpp 1.3.0. Machine B: Ubuntu 18.04, running ros2 foxy and rmw_fastrtps_cpp 1.3.0.

These machines are connected over a VPN, I'm using OpenVPN. I can ping machine A from machine B and vice-versa.

In both machines, the RMW implementation and the default profile is defined in the bashrc:

export  RMW_IMPLEMENTATION=rmw_fastrtps_cpp
export FASTRTPS_DEFAULT_PROFILES_FILE=~/develop_ws/imp_fastrtps_profile.xml

The imp_fastrtps_profile.xml for machine B is the following:

<?xml version="1.0" encoding="UTF-8" ?>
<dds>
    <profiles>
        <transport_descriptors>
            <transport_descriptor>
                <transport_id>veelpeers</transport_id>
                <type>UDPv4</type>
                <maxInitialPeersRange>10</maxInitialPeersRange>
            </transport_descriptor>
        </transport_descriptors>
        <participant profile_name="test_profile" is_default_profile="true">
            <rtps>
                <builtin>
                    <metatrafficUnicastLocatorList> 
                        <locator/> 
                    </metatrafficUnicastLocatorList>
                    <initialPeersList>
                        <locator>
                            <udpv4>
                                <address>127.0.0.1</address>
                            </udpv4>
                        </locator>
                        <locator>
                            <udpv4>
                                <address>MACHINE_A_OPENVPN_IP</address>
                            </udpv4>
                        </locator>
                    </initialPeersList>
                </builtin>
                <userTransports>
                <transport_id>veelpeers</transport_id>
                </userTransports>
                <useBuiltinTransports>false</useBuiltinTransports>
            </rtps>
        </participant>
        <publisher profile_name="test_publisher_profile" is_default_profile="true">
            <qos>
                <reliability>
                    <kind>BEST_EFFORT</kind>
                </reliability>
            </qos>
        </publisher>
        <subscriber profile_name="test_subscription_profile" is_default_profile="true">
            <qos>
                <reliability>
                    <kind>BEST_EFFORT</kind>
                </reliability>
            </qos>
        </subscriber>
    </profiles>
</dds>

The imp_fastrtps_profile.xml for machine A is the following:

<?xml version="1.0" encoding="UTF-8" ?>
<dds>
    <profiles>
        <transport_descriptors>
            <transport_descriptor>
                <transport_id>veelpeers</transport_id>
                <type>UDPv4</type>
                <maxInitialPeersRange>10</maxInitialPeersRange>
            </transport_descriptor>
        </transport_descriptors>
        <participant profile_name="test_profile" is_default_profile="true">
            <rtps>
                <builtin>
                    <metatrafficUnicastLocatorList> 
                        <locator/> 
                    </metatrafficUnicastLocatorList>
                    <initialPeersList>
                        <locator>
                            <udpv4>
                                <address>127.0.0.1</address>
                            </udpv4>
                        </locator>
                        <locator>
                            <udpv4>
                                <address>MACHINE_B_OPENVPN_IP</address>
                            </udpv4>
                        </locator>
                    </initialPeersList>
                </builtin>
                <userTransports>
                <transport_id>veelpeers</transport_id>
                </userTransports>
                <useBuiltinTransports>false</useBuiltinTransports>
            </rtps>
        </participant>
        <publisher profile_name="test_publisher_profile" is_default_profile="true">
            <qos>
                <reliability>
                    <kind>BEST_EFFORT</kind>
                </reliability>
            </qos>
        </publisher>
        <subscriber profile_name="test_subscription_profile" is_default_profile="true">
            <qos>
                <reliability>
                    <kind>BEST_EFFORT</kind>
                </reliability>
            </qos>
        </subscriber>
    </profiles>
</dds>

When my machine B is connected to my Wifi hotspot, machine A and machine B can communicate without any problems, which leads me to believe that the configuration file is correct. When I connect my machine B to the modem that we have in the vehicle, machine A and machine B can't communicate anymore. I'm using a Teltonika RUTX12 in the vehicle with a load balancing script to divide the load between two SIM cards. This setup works with CycloneDDS but I would like to try with FastRTPS.

Any help is very much appreciated. Cheers, Filipe

Bug report

Required Info:

• Operating System:
  • Ubuntu 20.04
• Installation type:
  • Binaries
• Version or commit hash:
  • rmw_fastrtps_cpp : required=1.3.0, local=1.3.0
• DDS implementation:
  • Fast-RTPS
• Client library (if applicable):
  • rclcpp rclpy

[fujitatomoya]

These machines are connected over a VPN, I'm using OpenVPN. I can ping machine A from machine B and vice-versa.

this means that you can ping bi-direction even if machine B is behind the modem that you have in the vehicle, right?

@EduPonz any thoughts?

[EduPonz]

Its difficult to tell with the information provided. It might have to do with VPN configuration, routing rules, firewalls, etc. Would it be possible to see some Wireshark captures from both machines? I'd like to verify that the announced IPs are correct, and that the packages are in fact being sent using the correct interface in each machine.

I'm also thinking that it might be useful to set an interface whitelist (XML config here) including the VPN interface and localhost, hence avoiding using the remaining interfaces. I do have one question though, is the VPN interface up and running prior to running the applications? I'm also wondering why including this bit:

<metatrafficUnicastLocatorList> 
    <locator/> 
</metatrafficUnicastLocatorList>

This is an scenario which we ourselves run in a home-office setup, so I'm sure it's just a configuration matter somewhere.

[FiIipe]

@fujitatomoya Yes, I can ping bi-direction, even if machine B is behind the modem that I have in the vehicle.

Update: I found out that it only stops working when the load balancing script is running. Otherwise it runs just fine. This script divides the load that we have between two 4G connections to deal with lack of coverage and bandwidth issues.

@EduPonz

• Yes, I can provide wireshark captures, I will do it as soon as possible.
• Ok, we will try to set an interface whitelist. What is the difference between using the whitelist and defining the localhost IP and the OpenVPN IP in the initialPeersList?
• Yes, the VPN interface is up and running prior to running the applications.
• I don't remember why I included the metatrafficUnicastLocatorList, but if it is not necessary I can remove it.

Do you have any template for configuration of fast_RTPS XML file to use with OpenVPN?

Thank you very much for your help!

[EduPonz]

This script divides the load that we have between two 4G connections to deal with lack of coverage and bandwidth issues.

Does this mean that the source IP changes when switching from one to the other?

• What is the difference between using the whitelist and defining the localhost IP and the OpenVPN IP in the initialPeersList?

By default, Fast DDS will sent every packet through all the available interfaces; whitelisting limits this behaviour the the interfaces listed. On the other hand, the list of initial peers is used to send unicast announcements to those peers directly in addition to the default multicast announcements. Mind that these announcements are sent using either all the interfaces (default) or the ones specified in the whitelist (if configured).

• I don't remember why I included the metatrafficUnicastLocatorList, but if it is not necessary I can remove it.

I think it is not needed.

Do you have any template for configuration of fast_RTPS XML file to use with OpenVPN?

Unfortunately no, since we do not use OpenVPN for our VPN. In any case, I don't think there should be any configuration specific for it. The captures will most likely put us in the light, specially regarding the switching between 4G connections.

[javad123javad]

Hi, I'm Filipe's colleague. We have two OpenVPN interfaces running on ports 1195 and 1196. The 4G modem also has two 4G WAN interfaces. To reduce the loads on the WAN interfaces, we divide our traffic on the source side, then we send them over VPN interfaces. When using CyclonDDS, it works normally. Here I attach the load balancing script for your reference:

#!/bin/sh
sleep 10
echo 1 >| /proc/sys/net/ipv4/ip_forward
echo 0 >| /proc/sys/net/ipv4/conf/all/rp_filter

iptables -t mangle -Fi
iptables -t mangle -X

iptables -t mangle -A PREROUTING -i br-lan -p udp  -m udp -d 0.0.0.0/0 --dport 1196 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -i br-lan -p udp -m udp -d 0.0.0.0/0 --dport 1195 -j MARK --set-mark 1

# Add two new table to the ip_tables
if ! cat /etc/iproute2/rt_tables | grep -q '^250'
then
        echo '250     rt_link1' >> /etc/iproute2/rt_tables
fi
if ! cat /etc/iproute2/rt_tables | grep -q '^251'
then
        echo '251     rt_link2' >> /etc/iproute2/rt_tables
fi

ip route flush table rt_link1
# Map 1195 to wwan0
ip route add table rt_link1 default dev wwan0
ip route flush table rt_link2
# Map 1196 to wwan1
ip route add table rt_link2 default dev wwan1

ip rule del from all fwmark 2 2>/dev/null
ip rule del from all fwmark 1 2>/dev/null
ip rule add fwmark 1 table rt_link1
ip rule add fwmark 2 table rt_link2
ip route flush cache