MTU problem

[echolimazulu]

Hello Adrien,

I have problems with the MTU values ​​that Glorytun sets automatically when the path is raised. My network interfaces are using MTU 1500, but Glorytun, when entering the "path up" command, changes the MTU of the client tun0 interface and the server interface to MTU 1357. The glorytun show command displays the MTU value of 1357, the glorytun path command displays the MTU 1379. Manually change the MTU on the interface, after the "glorytun path" nothing is updated in the displayed glorytun metrics, the value changes only in ifconfig. I know that the values ​​should be 1450 and 1472, but in my basic setup, without any significant changes in the OS, these values ​​do not correspond to those expected when Glorytun is running in normal mode. Please, tell me what could be the problem and can I fix it on my own or is it a bug?

Specification: OS: Debian Buster Glorytun: binary release 0.3.4 Network Interfaces MTUs: 1500 Rate: auto or any fixed rate Cipher: any

glorytun show server tun0: pid: 3045 bind: 192.168.254.5 port 5000 mtu: 1357 cipher: chacha20poly1305 glorytun path path UP status: OK bind: 192.168.254.5 port 5000 public: 192.168.254.5 port 5000 peer: 192.168.254.6 port 5000 mtu: 1379 bytes rtt: 1107.740 ms rttvar: 2031.065 ms rate: auto losslim: 100 beat: 100 ms tx: rate: 0 bytes/sec loss: 0 percent total: 48 packets rx: rate: 0 bytes/sec loss: 0 percent total: 96 packets

[angt]

Hi, I see a very high RTT on your path, glorytun doesn't support >500ms path by default. This may be the cause of the wrong estimation of the MTU. I'll do some tests on my side to check but in the meantime you can maybe change the beat of the path at 500ms.

[echolimazulu]

Thank you for reply,

If i set Cipher aegis256 and use fixed rates is 35mbit for RX/TX:

glorytun show client tun0: pid: 18553 bind: 0.0.0.0 port 5000 peer: 192.168.254.5 port 5000 mtu: 1357 cipher: aegis256

glorytun path path UP status: OK bind: 192.168.254.6 port 5000 public: 192.168.254.6 port 5000 peer: 192.168.254.5 port 5000 mtu: 1379 bytes rtt: 30.958 ms rttvar: 11.406 ms rate: fixed losslim: 100 beat: 100 ms tx: rate: 4375000 bytes/sec loss: 0 percent total: 45 packets rx: rate: 4375000 bytes/sec loss: 0 percent total: 15 packets path UP status: OK bind: 192.168.254.10 port 5000 public: 192.168.254.10 port 5000 peer: 192.168.254.5 port 5000 mtu: 1379 bytes rtt: 28.932 ms rttvar: 9.240 ms rate: fixed losslim: 100 beat: 100 ms tx: rate: 4375000 bytes/sec loss: 0 percent total: 45 packets rx: rate: 4375000 bytes/sec loss: 0 percent total: 15 packets

ifconfig tun0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1357 inet 10.0.1.2 netmask 255.255.255.255 destination 10.0.1.1 unspec 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00 txqueuelen 500 (UNSPEC) RX packets 0 bytes 0 (0.0 B) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 0 bytes 0 (0.0 B) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens224: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.254.6 netmask 255.255.255.252 broadcast 192.168.254.7 ether 00:0c:29:84:9d:52 txqueuelen 1000 (Ethernet) RX packets 245751633 bytes 231318617728 (215.4 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 165587164 bytes 35605805040 (33.1 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

ens256: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 inet 192.168.254.10 netmask 255.255.255.252 broadcast 192.168.254.11 ether 00:0c:29:84:9d:5c txqueuelen 1000 (Ethernet) RX packets 95366442 bytes 90785044639 (84.5 GiB) RX errors 0 dropped 0 overruns 0 frame 0 TX packets 64375486 bytes 14254032682 (13.2 GiB) TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

[angt]

This looks more legit :) You can check/confirm the detected MTU with the command:

ping -M do -s <SIZE> -I <SRC> <DEST>

[echolimazulu]

For some reason 1450 and 1472 are not available to pass through my network interface (MTU1500 - SRCs 254.6, 254.10). Is this normal behavior? These interfaces are based on NAT (masquerade) for the LTE modem.

ping -M do -s 1357 -I 192.168.254.6 192.168.254.5 PING 192.168.254.5 (192.168.254.5) from 192.168.254.6 : 1357(1385) bytes of data. 1365 bytes from 192.168.254.5: icmp_seq=1 ttl=64 time=49.4 ms 1365 bytes from 192.168.254.5: icmp_seq=2 ttl=64 time=27.8 ms 1365 bytes from 192.168.254.5: icmp_seq=3 ttl=64 time=33.6 ms ^C --- 192.168.254.5 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 5ms rtt min/avg/max/mdev = 27.793/36.932/49.415/9.140 ms

ping -M do -s 1379 -I 192.168.254.6 192.168.254.5 PING 192.168.254.5 (192.168.254.5) from 192.168.254.6 : 1379(1407) bytes of data. 1387 bytes from 192.168.254.5: icmp_seq=1 ttl=64 time=30.4 ms 1387 bytes from 192.168.254.5: icmp_seq=2 ttl=64 time=38.4 ms 1387 bytes from 192.168.254.5: icmp_seq=3 ttl=64 time=45.1 ms ^C --- 192.168.254.5 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 5ms rtt min/avg/max/mdev = 30.428/37.972/45.058/5.981 ms

ping -M do -s 1450 -I 192.168.254.6 192.168.254.5 PING 192.168.254.5 (192.168.254.5) from 192.168.254.6 : 1450(1478) bytes of data. ^C --- 192.168.254.5 ping statistics --- 3 packets transmitted, 0 received, 100% packet loss, time 40ms

ping -M do -s 1472 -I 192.168.254.6 192.168.254.5 PING 192.168.254.5 (192.168.254.5) from 192.168.254.6 : 1472(1500) bytes of data. ^C --- 192.168.254.5 ping statistics --- 5 packets transmitted, 0 received, 100% packet loss, time 86ms

[angt]

No idea about your network setup, MTU is hard to guess and that's exactly why glorytun do it for you :)

[echolimazulu]

Thank you and thank Glorytun for help in that,

With these MTU settings, the throughput is at 28Mbit/s when using Glorytun and two uplinks. At the same time, the nominal throughput for each of the interfaces is 28-70Mbit/s. The latency ranges from 30-60ms. What could be the reason for such a low level of performance through Glorytun? Perhaps it makes sense to make additional settings, for example, in the queue size and MTU fragmentation does not significantly affect the final results? After Glorytun I am using a high performance tunnel over UDP.

[echolimazulu]

glorytun bench cipher: aegis256

size min mean max 20 869 Mbps 887 Mbps 897 Mbps 150 4681 Mbps 4703 Mbps 4712 Mbps 280 6935 Mbps 6950 Mbps 6960 Mbps 410 8070 Mbps 8138 Mbps 8151 Mbps 540 9113 Mbps 9160 Mbps 9187 Mbps 670 9594 Mbps 9829 Mbps 9867 Mbps 800 10477 Mbps 10569 Mbps 10590 Mbps 930 10693 Mbps 10738 Mbps 10756 Mbps 1060 10871 Mbps 11075 Mbps 11134 Mbps 1190 11376 Mbps 11443 Mbps 11476 Mbps 1320 11547 Mbps 11672 Mbps 11715 Mbps 1450 11901 Mbps 11944 Mbps 11971 Mbps

[angt]

Can you do iperf3 -u -b 100M -B <client tun0 ip> <server tun0 ip> ?

[echolimazulu]

iperf3 -u -b 100M -B 10.0.1.2 -c 10.0.1.1 Connecting to host 10.0.1.1, port 5201 [ 5] local 10.0.1.2 port 34215 connected to 10.0.1.1 port 5201 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 11.9 MBytes 99.9 Mbits/sec 9571 [ 5] 1.00-2.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 2.00-3.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 3.00-4.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 4.00-5.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 5.00-6.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 6.00-7.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 7.00-8.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 9.00-10.00 sec 11.9 MBytes 100 Mbits/sec 9579

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 119 MBytes 100 Mbits/sec 0.000 ms 0/95778 (0%) sender [ 5] 0.00-10.35 sec 72.6 MBytes 58.8 Mbits/sec 0.800 ms 37474/95774 (39%) receiver

iperf Done. iperf3 -u -b 100M -B 10.0.1.2 -c 10.0.1.1 Connecting to host 10.0.1.1, port 5201 [ 5] local 10.0.1.2 port 48657 connected to 10.0.1.1 port 5201 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 11.9 MBytes 99.9 Mbits/sec 9571 [ 5] 1.00-2.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 2.00-3.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 3.00-4.00 sec 11.9 MBytes 100 Mbits/sec 9582 [ 5] 4.00-5.00 sec 11.9 MBytes 100 Mbits/sec 9575 [ 5] 5.00-6.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 6.00-7.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 7.00-8.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 9.00-10.00 sec 11.9 MBytes 100 Mbits/sec 9579

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 119 MBytes 100 Mbits/sec 0.000 ms 0/95778 (0%) sender [ 5] 0.00-10.35 sec 70.9 MBytes 57.5 Mbits/sec 0.761 ms 38794/95764 (41%) receiver

iperf Done. iperf3 -u -b 100M -B 10.0.1.2 -c 10.0.1.1 Connecting to host 10.0.1.1, port 5201 [ 5] local 10.0.1.2 port 37972 connected to 10.0.1.1 port 5201 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 11.9 MBytes 99.9 Mbits/sec 9572 [ 5] 1.00-2.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 2.00-3.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 3.00-4.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 4.00-5.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 5.00-6.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 6.00-7.00 sec 11.9 MBytes 100 Mbits/sec 9579 [ 5] 7.00-8.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 8.00-9.00 sec 11.9 MBytes 100 Mbits/sec 9578 [ 5] 9.00-10.00 sec 11.9 MBytes 100 Mbits/sec 9579

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 119 MBytes 100 Mbits/sec 0.000 ms 0/95778 (0%) sender [ 5] 0.00-10.32 sec 70.1 MBytes 57.0 Mbits/sec 0.469 ms 39435/95778 (41%) receiver

[angt]

So i guess you have ~30Mbps on each path and glorytun gives you ~60 on UDP. It looks good so far. You can now test TCP (just iperf3 -B <IP> <IP>) to see if you get the same result or less.

[echolimazulu]

I set a fixed 30mbit for rx and tx for each path. rtt for paths 40-45ms Is it normal for TCP throughput to go from 5mbit to 60mbit continuously? When tested using MPTCP, iperf3 (tcp) produced up to 50-90mbit/s on two paths.

iperf3 -u -b 60M -B 10.0.1.2 -c 10.0.1.1 Connecting to host 10.0.1.1, port 5201 [ 5] local 10.0.1.2 port 59769 connected to 10.0.1.1 port 5201 [ ID] Interval Transfer Bitrate Total Datagrams [ 5] 0.00-1.00 sec 7.15 MBytes 60.0 Mbits/sec 5743 [ 5] 1.00-2.00 sec 7.15 MBytes 60.0 Mbits/sec 5747 [ 5] 2.00-3.00 sec 7.15 MBytes 60.0 Mbits/sec 5747 [ 5] 3.00-4.00 sec 7.15 MBytes 60.0 Mbits/sec 5747 [ 5] 4.00-5.00 sec 7.15 MBytes 60.0 Mbits/sec 5748 [ 5] 5.00-6.00 sec 7.15 MBytes 60.0 Mbits/sec 5746 [ 5] 6.00-7.00 sec 7.15 MBytes 60.0 Mbits/sec 5748 [ 5] 7.00-8.00 sec 7.15 MBytes 60.0 Mbits/sec 5747 [ 5] 8.00-9.00 sec 7.15 MBytes 60.0 Mbits/sec 5747 [ 5] 9.00-10.00 sec 7.15 MBytes 60.0 Mbits/sec 5747

---

[ ID] Interval Transfer Bitrate Jitter Lost/Total Datagrams [ 5] 0.00-10.00 sec 71.5 MBytes 60.0 Mbits/sec 0.000 ms 0/57467 (0%) sender [ 5] 0.00-10.32 sec 67.7 MBytes 55.0 Mbits/sec 0.521 ms 2982/57385 (5.2%) receiver

iperf Done. iperf3 -B 10.0.1.2 -c 10.0.1.1 -t 50 Connecting to host 10.0.1.1, port 5201 [ 5] local 10.0.1.2 port 42575 connected to 10.0.1.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 472 KBytes 3.86 Mbits/sec 6 22.9 KBytes [ 5] 1.00-2.00 sec 777 KBytes 6.37 Mbits/sec 1 38.2 KBytes [ 5] 2.00-3.00 sec 1.26 MBytes 10.6 Mbits/sec 0 56.1 KBytes [ 5] 3.00-4.00 sec 1.58 MBytes 13.3 Mbits/sec 0 73.9 KBytes [ 5] 4.00-5.00 sec 2.19 MBytes 18.4 Mbits/sec 0 90.5 KBytes [ 5] 5.00-6.00 sec 1.78 MBytes 14.9 Mbits/sec 60 54.8 KBytes [ 5] 6.00-7.00 sec 1.65 MBytes 13.8 Mbits/sec 0 79.0 KBytes [ 5] 7.00-8.00 sec 2.01 MBytes 16.9 Mbits/sec 0 93.0 KBytes [ 5] 8.00-9.00 sec 2.20 MBytes 18.4 Mbits/sec 5 105 KBytes [ 5] 9.00-10.00 sec 2.13 MBytes 17.9 Mbits/sec 11 87.9 KBytes [ 5] 10.00-11.00 sec 2.26 MBytes 18.9 Mbits/sec 0 102 KBytes [ 5] 11.00-12.00 sec 2.50 MBytes 21.0 Mbits/sec 0 117 KBytes [ 5] 12.00-13.00 sec 2.74 MBytes 23.0 Mbits/sec 0 133 KBytes [ 5] 13.00-14.00 sec 3.35 MBytes 28.1 Mbits/sec 0 149 KBytes [ 5] 14.00-15.00 sec 3.78 MBytes 31.7 Mbits/sec 0 164 KBytes [ 5] 15.00-16.00 sec 3.72 MBytes 31.2 Mbits/sec 0 180 KBytes [ 5] 16.00-17.00 sec 3.96 MBytes 33.3 Mbits/sec 0 195 KBytes [ 5] 17.00-18.00 sec 4.27 MBytes 35.8 Mbits/sec 0 209 KBytes [ 5] 18.00-19.00 sec 4.88 MBytes 40.7 Mbits/sec 0 224 KBytes [ 5] 19.00-20.00 sec 5.00 MBytes 42.2 Mbits/sec 14 237 KBytes [ 5] 20.00-21.00 sec 5.24 MBytes 44.0 Mbits/sec 11 251 KBytes [ 5] 21.00-22.00 sec 4.94 MBytes 41.4 Mbits/sec 0 265 KBytes [ 5] 22.00-23.00 sec 5.49 MBytes 46.0 Mbits/sec 0 279 KBytes [ 5] 23.00-24.00 sec 5.98 MBytes 50.1 Mbits/sec 0 293 KBytes [ 5] 24.00-25.00 sec 5.67 MBytes 47.6 Mbits/sec 0 306 KBytes [ 5] 25.00-26.00 sec 5.79 MBytes 48.6 Mbits/sec 0 335 KBytes [ 5] 26.00-27.00 sec 6.40 MBytes 53.7 Mbits/sec 0 394 KBytes [ 5] 27.00-28.00 sec 6.34 MBytes 53.2 Mbits/sec 88 423 KBytes [ 5] 28.00-29.00 sec 5.49 MBytes 46.0 Mbits/sec 152 428 KBytes [ 5] 29.00-30.00 sec 6.10 MBytes 51.2 Mbits/sec 117 431 KBytes [ 5] 30.00-31.00 sec 6.10 MBytes 51.2 Mbits/sec 205 436 KBytes [ 5] 31.00-32.00 sec 6.04 MBytes 50.6 Mbits/sec 68 442 KBytes [ 5] 32.00-33.00 sec 6.59 MBytes 55.3 Mbits/sec 58 450 KBytes [ 5] 33.00-34.00 sec 4.94 MBytes 41.4 Mbits/sec 542 398 KBytes [ 5] 34.00-35.00 sec 6.10 MBytes 51.2 Mbits/sec 311 451 KBytes [ 5] 35.00-36.00 sec 6.16 MBytes 51.7 Mbits/sec 150 455 KBytes [ 5] 36.00-37.00 sec 6.10 MBytes 51.2 Mbits/sec 105 459 KBytes [ 5] 37.00-38.00 sec 6.28 MBytes 52.7 Mbits/sec 0 469 KBytes [ 5] 38.00-39.00 sec 5.79 MBytes 48.6 Mbits/sec 384 363 KBytes [ 5] 39.00-40.00 sec 6.53 MBytes 54.7 Mbits/sec 0 409 KBytes [ 5] 40.00-41.00 sec 5.79 MBytes 48.6 Mbits/sec 84 417 KBytes [ 5] 41.00-42.00 sec 6.28 MBytes 52.7 Mbits/sec 43 424 KBytes [ 5] 42.00-43.00 sec 6.59 MBytes 55.3 Mbits/sec 32 433 KBytes [ 5] 43.00-44.00 sec 6.53 MBytes 54.7 Mbits/sec 8 442 KBytes [ 5] 44.00-45.00 sec 6.53 MBytes 54.7 Mbits/sec 0 452 KBytes [ 5] 45.00-46.00 sec 6.46 MBytes 54.2 Mbits/sec 46 459 KBytes [ 5] 46.00-47.00 sec 6.46 MBytes 54.2 Mbits/sec 6 466 KBytes [ 5] 47.00-48.00 sec 6.59 MBytes 55.2 Mbits/sec 0 477 KBytes [ 5] 48.00-49.00 sec 6.10 MBytes 51.2 Mbits/sec 25 422 KBytes [ 5] 49.00-50.00 sec 6.71 MBytes 56.3 Mbits/sec 6 493 KBytes

---

[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-50.00 sec 235 MBytes 39.4 Mbits/sec 2538 sender [ 5] 0.00-50.08 sec 234 MBytes 39.2 Mbits/sec receiver

[angt]

No it's not normal. When doing the iperf, keep an eye on glorytun path to see if the RTT goes up and if it does, lower tx/rx again. Also, to fix TCP I generally recommand to use cake both sides:

tc qdisc replace dev tun0 root cake
ip link set tun0 txqlen 100

It's not magical either, but it usually helps a lot.

[angt]

It's looking bad at the beginning, TCP needs some time to understand how it can use the glorytun link. At the end you have a not so bad working aggregation. You can also try iperf3 [...] --congestion bbr to see if it's help. If it's better you can ask your system to use it by default.

[echolimazulu]

Currently I am using "cubic" on the client and on the server.

sudo tc qdisc replace dev tun0 root cake Error: Specified qdisc not found.

[angt]

Hmm, this where OverTheBox shines, It will do all this stuff for you :)

[echolimazulu]

--congestion bbr + ip link set tun0 txqlen 100 = fixed TCP

Result: iperf3 -B 10.0.1.2 -c 10.0.1.1 Connecting to host 10.0.1.1, port 5201 [ 5] local 10.0.1.2 port 39089 connected to 10.0.1.1 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 7.49 MBytes 62.8 Mbits/sec 571 398 KBytes [ 5] 1.00-2.00 sec 5.00 MBytes 41.9 Mbits/sec 996 451 KBytes [ 5] 2.00-3.00 sec 6.25 MBytes 52.4 Mbits/sec 26 390 KBytes [ 5] 3.00-4.00 sec 7.50 MBytes 62.9 Mbits/sec 6 428 KBytes [ 5] 4.00-5.00 sec 6.25 MBytes 52.4 Mbits/sec 0 413 KBytes [ 5] 5.00-6.00 sec 6.25 MBytes 52.4 Mbits/sec 27 418 KBytes [ 5] 6.00-7.00 sec 6.25 MBytes 52.4 Mbits/sec 0 405 KBytes [ 5] 7.00-8.00 sec 6.25 MBytes 52.4 Mbits/sec 0 380 KBytes [ 5] 8.00-9.00 sec 6.25 MBytes 52.4 Mbits/sec 0 451 KBytes [ 5] 9.00-10.00 sec 6.25 MBytes 52.4 Mbits/sec 3 377 KBytes

---

[ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 63.7 MBytes 53.5 Mbits/sec 1629 sender [ 5] 0.00-10.06 sec 60.8 MBytes 50.7 Mbits/sec receiver

iperf Done.

[angt]

Nice, I'm closing the issue as there was no MTU issue and TCP link aggregation works :)