search

Ysurac / openmptcprouter

OpenMPTCProuter is an open source solution to aggregate multiple internet connections using Multipath TCP (MPTCP) on OpenWrt
https://www.openmptcprouter.com/
GNU General Public License v3.0
1.8k stars 258 forks source link

glorytun throughput is tooo slow #2648

Closed rdmitry0911 closed 1 year ago

rdmitry0911 commented 1 year ago

Expected Behavior

Throughput via glorytun (default vpn for all protocols other than tcp) is about the same as with a regular tcp networking

Current Behavior

Throughput via glorytun is about 10 times less than a regular tcp networking->

Specifications

I have 3 internet lines: mobile, Starlink and fixed. All 3 are working good. I'm using the default configuration for everything. All the specifics is located only in /etc/config/network and in OMR bypass (both are attached). Nothing special in log files. However tracepath to 8.8.8.8 from OMR looks like this:

root@OpenMPTCProuter:~# tracepath 8.8.8.8
 1?: [LOCALHOST]                      pmtu 1500
 1:  10.255.255.1                                        1050.824ms 
 1:  10.255.255.1                                        2730.835ms 
 2:  192.168.40.1                                        2686.788ms 
 2:  192.168.40.1                                        1754.503ms 
 2:  192.168.40.1                                        778.461ms 
 3:  172.16.44.1                                         1925.886ms 
 4:  static.241.19.21.65.clients.your-server.de          1921.119ms 
 5:  213-239-224-125.clients.your-server.de              2410.352ms 
 6:  core40.sto.hetzner.com                              4663.999ms 
 6:  core40.sto.hetzner.com                              5458.358ms 
 7:  142.250.161.204                                     5774.685ms asymm 11 
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
12:  no reply
^C
root@OpenMPTCProuter:~#

while from VPS it looks like this:

root@omr-vps:~# tracepath 8.8.8.8
 1?: [LOCALHOST]                      pmtu 1500
 1:  _gateway                                              0.194ms 
 1:  _gateway                                              0.136ms 
 2:  172.16.44.1                                           0.230ms 
 3:  static.241.19.21.65.clients.your-server.de            0.565ms 
 4:  hos-tr1.juniper1.dc5.fsn1.hetzner.com                 1.049ms 
 5:  core40.sto.hetzner.com                                6.251ms 
 6:  142.250.161.204                                       6.455ms asymm 10 
 7:  no reply
 8:  no reply
 9:  no reply
10:  no reply
11:  no reply
^C
root@omr-vps:~#

Tracepath via individual internet lines also looks fine:

Screen Shot 2022-11-08 at 1 24 26 PM

Strange things on this picture are:

  1. The same wan address 192.168.40.1 for all the individual lines, which is the address of a router after VPS
  2. Wlan address 172.16.66.1, while it is definitely 10.11.11.1

/etc/config/network:

root@OpenMPTCProuter:~# cat /etc/config/network

config interface 'loopback'
    option proto 'static'
    option ipaddr '127.0.0.1'
    option netmask '255.0.0.0'
    option multipath 'off'
    option device 'lo'
    option metric '21'

config globals 'globals'
    option ula_prefix 'fdd3:316a:5c48::/48'
    option persistent_nic '1'
    option multipath 'enable'
    option mptcp_path_manager 'fullmesh'
    option mptcp_scheduler 'blest'
    option congestion 'cubic'
    option mptcp_checksum '0'
    option mptcp_debug '0'
    option mptcp_syn_retries '2'
    option mptcp_subflows '3'
    option mptcp_add_addr_accepted '1'
    option mptcp_add_addr_timeout '120'
    option mptcp_fullmesh_num_subflows '1'
    option mptcp_fullmesh_create_on_err '1'
    option mptcp_ndiffports_num_subflows '1'

config interface 'docker'
    option device 'docker0'
    option proto 'none'
    option auto '0'
    option multipath 'off'
    option metric '22'

config device
    option type 'bridge'
    option name 'docker0'

config interface 'lan'
    option device 'eth0'
    option ifname 'eth0'
    option proto 'static'
    option netmask '255.255.255.0'
    option ipaddr '172.16.66.1'
    option ipv6 '0'
    option delegate '0'
    option addlatency '0'
    option multipath 'off'
    option ip4table 'lan'
    option label 'lan'
    option defaultroute '0'
    option peerdns '0'
    option metric '23'

config interface 'mqtt'
    option device '@lan'
    option proto 'static'
    option netmask '255.255.255.0'
    option ipaddr '172.16.66.100'
    option multipath 'off'
    option metric '24'

config rule 'lan_rule'
    option lookup 'lan'
    option priority '100'

config interface 'wan1'
    option ip4table 'wan'
    option device 'etha6fd3e'
    option mac_original '00:e0:4c:a6:fd:3e'
    option proto 'static'
    option label 'Marathi'
    option ipv6 '0'
    option defaultroute '0'
    option peerdns '0'
    option ipaddr '192.168.31.15'
    option netmask '255.255.255.0'
    option gateway '192.168.31.1'
    option metric '17'
    option multipath 'master'

config interface 'wan2'
    option ip4table 'wan'
    option device 'ethf0c1c5'
    option mac_original '7e:a1:ae:f0:c1:c5'
    option proto 'dhcp'
    option ipv6 '0'
    option defaultroute '0'
    option peerdns '0'
    option label 'iPhone 11 Pro Max'
    option metric '18'
    option multipath 'off'

config interface 'wan3'
    option ip4table 'wan'
    option defaultroute '0'
    option ipv6 '0'
    option peerdns '0'
    option proto 'dhcp'
    option device 'eth681b5c'
    option mac_original '00:e0:4c:68:1b:5c'
    option label 'Starlink'
    option metric '19'
    option multipath 'on'

config interface 'wan4'
    option proto 'static'
    option type 'macvlan'
    option device 'wan4'
    option masterintf 'eth0'
    option ip4table 'lan'
    option defaultroute '0'
    option peerdns '0'
    option label 'line1'
    option ipv6 '0'
    option ipaddr '10.22.0.10'
    option netmask '255.255.255.0'
    option gateway '10.22.0.1'
    option metric '29'
    option multipath 'off'

config interface 'wan5'
    option proto 'static'
    option type 'macvlan'
    option device 'wan5'
    option masterintf 'eth0'
    option ip4table 'lan'
    option defaultroute '0'
    option ipv6 '0'
    option peerdns '0'
    option label 'line2'
    option ipaddr '10.23.0.10'
    option netmask '255.255.255.0'
    option gateway '10.23.0.1'
    option metric '30'
    option multipath 'on'

config interface 'wlan'
    option proto 'static'
    option ip4table 'lan'
    option defaultroute '0'
    option peerdns '0'
    option ipv6 '0'
    option label 'Wlan_for_urgent_connection'
    option multipath 'off'
    option ipaddr '10.11.11.1'
    option netmask '255.255.255.0'
    option metric '25'
    option addlatency '0'
    option device 'wlan0'

config interface 'wg'
    option proto 'wireguard'
    option private_key 'XXXXXXX'
    list addresses '10.90.10.34/32'
    option nohostroute '1'
    option multipath 'off'
    option ipv6 '0'
    option defaultroute '0'
    option peerdns '0'
    option metric '26'

config wireguard_wg 'wgserver'
    option public_key 'XXXXXXX'
    option endpoint_host 'XXXXXXX'
    option endpoint_port 'XXXXXX'
    option route_allowed_ips '1'
    option persistent_keepalive '25'
    list allowed_ips '10.90.10.0/24'
    list allowed_ips '10.90.13.0/24'
    list allowed_ips '172.16.44.0/24'
    list allowed_ips '10.30.0.0/24'
    list allowed_ips '10.45.0.0/24'
    list allowed_ips '10.20.251.0/24'
    list allowed_ips '192.168.51.0/24'
    list allowed_ips '172.16.56.0/24'

config device 'wan1_dev'
    option name 'etha6fd3e'
    option txqueuelen '20'

config device 'wan2_dev'
    option name 'ethf0c1c5'
    option txqueuelen '20'

config device 'wan3_dev'
    option txqueuelen '20'
    option name 'eth681b5c'

config device 'wan4_dev'
    option type 'macvlan'
    option mode 'vepa'
    option ifname 'eth0'
    option name 'wan4'
    option txqueuelen '20'

config device 'wan5_dev'
    option type 'macvlan'
    option mode 'vepa'
    option ifname 'eth0'
    option name 'wan5'
    option txqueuelen '20'

config interface 'omrvpn'
    option device 'tun0'
    option ip4table 'vpn'
    option multipath 'off'
    option leasetime '12h'
    option type 'tunnel'
    option txqueuelen '100'
    option metric '1200'
    option proto 'none'

config interface 'omr6in4'
    option proto '6in4'
    option ip4table 'vpn'
    option multipath 'off'
    option ipaddr '10.255.255.2'
    option peeraddr '10.255.255.1'
    option auto '0'
    option metric '1201'
    option ip6addr 'fe80::a00:2/126'
    option gateway 'fe80::a00:1/126'

config device 'lan_dev'
    option name 'eth0'

config interface 'freeswitch'
    option proto 'none'
    option auto '1'
    option device 'tun1'
    option multipath 'off'
    option metric '35'
    option ipv6 '0'
    option defaultroute '0'
    option peerdns '0'

config interface 'openwrt'
    option proto 'none'
    option auto '1'
    option device 'tap0'
    option multipath 'off'
    option metric '36'
    option ipv6 '0'
    option defaultroute '0'
    option peerdns '0'

config device
    option type 'macvlan'
    option ifname 'etha6fd3e'
    option mode 'vepa'
    option name 'etha6fd3emac0'
    option ipv6 '0'

config interface 'ubnt_default'
    option proto 'static'
    option device 'etha6fd3emac0'
    option ipaddr '192.168.1.21'
    option netmask '255.255.255.0'
    option multipath 'off'
    option addlatency '0'
    option metric '38'

root@OpenMPTCProuter:~#

/etc/config/omr-bypas

root@OpenMPTCProuter:~# cat /etc/config/omr-bypass 

config interface 'all'

config proto 'm6replay'
    list url 'm6web.fr'
    list url '6play.fr'
    list url '6cloud.fr'

config proto 'mycanal'
    list url 'mycanal.fr'
    list url 'canal-plus.com'
    list url 'canalplus.com'
    list url 'canalplus-cdn.net'
    list url 'canalplus.pro'
    list url 'canal-plus.net'

config proto 'minecraft'
    list url 'authserver.mojang.com'

config proto 'lesnumeriques'
    list url 'lesnumeriques.com'
    list url 'botscorner.com'
    list url 'app.botscorner.com'

config proto 'disneyplus'
    list url 'bamgrid.com'
    list url 'disney-plus.net'

config proto 'amazonvideo'
    list url 'cloudfront.net'
    list url 'llnw.net'

config interface 'lo'
    option id '21'

config interface 'eth0'
    option id '24'

config interface 'wan1'
    option id '3'

config interface 'wan2'
    option id '12'

config interface 'tun0'
    option id '1200'

config interface 'wlan0'
    option id '25'

config interface 'eth1'
    option id '13'

config interface 'eth2'
    option id '10'

config interface 'eth681B5C'
    option id '9'

config interface 'ethf0c1c5'
    option id '18'

config interface 'ethf0c1c6'
    option id '12'

config interface 'ethBE236F'
    option id '12'

config interface 'eth681b5c'
    option id '19'

config interface 'ethbe236f'
    option id '12'

config interface 'wg'
    option id '26'

config interface 'docker0'
    option id '22'

config interface 'etha6fd3e'
    option id '17'

config interface 'eth0044bf'
    option id '9'

config interface 'wan3'
    option id '13'

config interface 'wan4'
    option id '29'

config interface 'wan5'
    option id '30'

config interface 'wan6'
    option id '22'

config interface 'wlan'
    option id '25'

config interface 'ethfed489'
    option id '13'

config interface 'eth6b1b5c'
    option id '19'

config domains
    option note 'Aegean Air'
    option name 'aegeanair.com'
    option interface 'etha6fd3e'

config interface 'tun1'
    option id '35'

config interface 'tap0'
    option id '36'

config domains
    option name 'nbg.gr'
    option interface 'ethf0c1c5'
    option note 'NBG (Bank)'

config domains
    option name 'winbank.gr'
    option note 'Piraeus Bank'
    option interface 'etha6fd3e'

config domains
    option name '17track.net'
    option note '17track.net'
    option enabled '0'
    option interface 'ethf0c1c5'

config domains
    option name '*.gr'
    option interface 'ethf0c1c5'
    option note 'All greece'

config domains
    option name 'wind.gr'
    option interface 'ethf0c1c5'
    option note 'Wind'

config domains
    option name 'cosmote.gr'
    option interface 'etha6fd3e'

config domains
    option name 'forum.xda-developers.com'
    option interface 'eth681b5c'

config interface 'etha6fd3emac0'
    option id '38'

config dest_port
    option dport 'XXXXX'
    option proto 'udp'
    option interface 'eth681b5c'
    option note 'wg0 via starlink'

config domains
    option name 'google.com'
    option interface 'etha6fd3e'
    option note 'accounts.google.com'

config lan_ip
    option ip '172.16.66.120'
    option interface 'etha6fd3e'
    option note 'lg gram'
rdmitry0911 commented 1 year ago

I noticed a strange records in system log:

Tue Nov  8 14:36:39 2022 daemon.debug dnsmasq-script[1]: /usr/lib/dnsmasq/dhcp-script.sh: .: line 5: can't open '/usr/share/libubox/jshn.sh': No such file or directory
Tue Nov  8 14:36:39 2022 daemon.warn dnsmasq[1]: script process exited with status 2

However, /usr/share/libubox/jshn.sh is there with correct access rights:

root@OpenMPTCProuter:~# ls -l  /usr/share/libubox/jshn.sh
-rwxr-xr-x    1 root     root          5457 May 18 01:11 /usr/share/libubox/jshn.sh
root@OpenMPTCProuter:~#

I have no idea if it is related to weak glorytun throughput issue, but it is abnormal too

Ysurac commented 1 year ago

Your configuration is strange with some macvlan, some direct,... You should start a fresh configuration. In the tracepath you have "192.168.40.1" multiple times. The IP shown in status pages for wan is the result of a request to the VPN so this is the IP of the router shown, should not be a big problem.

rdmitry0911 commented 1 year ago

Your configuration is strange with some macvlan, some direct,...

It is because I have 2 lines on my roof with a direct connection and mobile from iphone I have indoors connected to a lan host.

In the tracepath you have "192.168.40.1" multiple times.

It's only with glorytun tcp vpn. With glorytun udp it is totally different:

root@OpenMPTCProuter:~# tracepath -n 9.9.9.9
 1?: [LOCALHOST]                      pmtu 1442
 1:  10.255.254.1                                        105.676ms 
 1:  10.255.254.1                                         84.861ms 
 2:  192.168.40.1                                        102.696ms 
 3:  172.16.44.1                                          83.862ms 
 4:  65.21.19.241                                         87.072ms 
 5:  213.239.224.125                                     112.591ms 
 6:  213.239.203.210                                      80.378ms 
 7:  185.1.160.109                                       137.729ms 
 8:  109.200.218.46                                      129.955ms 
 9:  188.122.80.228                                      122.205ms 
10:  9.9.9.9                                             122.031ms !H
     Resume: pmtu 1442 
root@OpenMPTCProuter:~#

It looks good and pmtu is different.

Maybe mtu with glorytun tcp is calculated wrong way and this wrong mtu is a reason for the problem?

And a question about /usr/share/libubox/jshn.sh May it be tied to this problem, or it is a different bug and I have to open another issue?

Ysurac commented 1 year ago

You can try to set a lowest MTU on tun0 and check: ip link set mtu 1440 dev tun0 via SSH on the router.

rdmitry0911 commented 1 year ago

You can try to set a lowest MTU on tun0 and check: ip link set mtu 1440 dev tun0

I did a try. No help.

You should start a fresh configuration.

Is there a way to start over without a factory reset as I have some packages installed and I don't want to reinstall them all?

Ysurac commented 1 year ago

No but you can save installed package list when you do a backup.

rdmitry0911 commented 1 year ago

No but you can save installed package list when you do a backup.

That's strange. Everything is written in config files. The packages itself are the regular ones. Why don't remove the config files related to the omr configuration leaving all the rest files in places? I'm going to reinitialize omr remotely without having direct access to the system and make a factory reset might leave me without an access to the system. Clearing config files would be much more safe procedure in this case.

github-actions[bot] commented 1 year ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days