search

M0dEx / quincy

QUIC-based VPN
MIT License
84 stars 7 forks source link

Throughput issues with `datagram too large` errors #40

Closed vukitoso closed 3 months ago

vukitoso commented 4 months ago

Hello. Is your QUIC channel throughput greatly reduced? I have a VPS with a limit of 10 Mbps.

iperf3 -c *.*.*.* -P10
[SUM]   0.00-10.00  sec  15.9 MBytes  13.3 Mbits/sec  531             sender
[SUM]   0.00-11.11  sec  11.9 MBytes  8.97 Mbits/sec                  receiver

But through the tunnel the speed drops by 5-10 times:

-P10
[SUM]   0.00-10.00  sec  2.59 MBytes  2.17 Mbits/sec   22             sender
[SUM]   0.00-15.93  sec  1.83 MBytes   966 Kbits/sec                  receiver

-P10
[SUM]   0.00-10.00  sec  1.43 MBytes  1.20 Mbits/sec   27             sender
[SUM]   0.00-16.35  sec   918 KBytes   460 Kbits/sec                  receiver

-P3
[SUM]   0.00-10.00  sec  1.56 MBytes  1.31 Mbits/sec   16             sender
[SUM]   0.00-11.70  sec  1.24 MBytes   892 Kbits/sec                  receiver

-P1
[  5]   0.00-10.00  sec  1.71 MBytes  1.44 Mbits/sec    8             sender
[  5]   0.00-11.49  sec  1.48 MBytes  1.08 Mbits/sec                  receiver
M0dEx commented 4 months ago

Hi!

Could you please specify what OS is the VPS running, what specs it has and what are the properties of the connection (mainly MTU)?

vukitoso commented 4 months ago

Could you please specify what OS is the VPS running, what specs it has and what are the properties of the connection (mainly MTU)?

Server VPS x86_64: Debian 12 CPU: Limit 30% from 2300 MHz (=690 MHz) RAM: 512Mb eth0: mtu 1500 tun0: mtu 1400 Internet speed is limited to exactly 10 Mbit/s

Client ARMv7 board Debian 12 4 x 1500 MHz RAM: 1024Mb eth0: mtu 1500 tun0: mtu 1400 Internet speed up to approximately 50 Mbit/s

M0dEx commented 4 months ago

Did you use the provided binaries or did you compile Quincy yourself (please answer for both the server and the client)?

What version of Quincy (I assume latest - 0.6.1)?

My initial hypothesis is that the very limited CPU on the VPS is causing the performance issues. Did you check the CPU load during the iperf run? What was it?

vukitoso commented 4 months ago

Did you use the provided binaries or did you compile Quincy yourself (please answer for both the server and the client)?

I collected it myself

cargo install quincy

quincy-0.6.1

The CPU is not heavily loaded: quincy-server process: 3-5% CPU: 3-14%

I noticed that when testing through iperf3 the ping increases greatly to 10.0.0.1

$ ping 10.0.0.1
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=72.6 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=66.9 ms
64 bytes from 10.0.0.1: icmp_seq=3 ttl=64 time=90.4 ms
64 bytes from 10.0.0.1: icmp_seq=4 ttl=64 time=69.6 ms
64 bytes from 10.0.0.1: icmp_seq=5 ttl=64 time=70.4 ms
64 bytes from 10.0.0.1: icmp_seq=6 ttl=64 time=67.4 ms
64 bytes from 10.0.0.1: icmp_seq=7 ttl=64 time=71.9 ms
64 bytes from 10.0.0.1: icmp_seq=8 ttl=64 time=65.5 ms
64 bytes from 10.0.0.1: icmp_seq=9 ttl=64 time=66.1 ms
64 bytes from 10.0.0.1: icmp_seq=10 ttl=64 time=67.6 ms
64 bytes from 10.0.0.1: icmp_seq=11 ttl=64 time=1145 ms
64 bytes from 10.0.0.1: icmp_seq=12 ttl=64 time=3208 ms
64 bytes from 10.0.0.1: icmp_seq=13 ttl=64 time=4226 ms
64 bytes from 10.0.0.1: icmp_seq=14 ttl=64 time=5510 ms
64 bytes from 10.0.0.1: icmp_seq=15 ttl=64 time=5660 ms
64 bytes from 10.0.0.1: icmp_seq=16 ttl=64 time=6128 ms
64 bytes from 10.0.0.1: icmp_seq=17 ttl=64 time=6487 ms
64 bytes from 10.0.0.1: icmp_seq=18 ttl=64 time=7763 ms
64 bytes from 10.0.0.1: icmp_seq=19 ttl=64 time=7814 ms
64 bytes from 10.0.0.1: icmp_seq=20 ttl=64 time=7724 ms
64 bytes from 10.0.0.1: icmp_seq=21 ttl=64 time=7986 ms
64 bytes from 10.0.0.1: icmp_seq=22 ttl=64 time=8167 ms
64 bytes from 10.0.0.1: icmp_seq=23 ttl=64 time=8206 ms
64 bytes from 10.0.0.1: icmp_seq=24 ttl=64 time=8208 ms
64 bytes from 10.0.0.1: icmp_seq=25 ttl=64 time=8124 ms
64 bytes from 10.0.0.1: icmp_seq=26 ttl=64 time=8135 ms
64 bytes from 10.0.0.1: icmp_seq=27 ttl=64 time=8224 ms
64 bytes from 10.0.0.1: icmp_seq=28 ttl=64 time=8511 ms
64 bytes from 10.0.0.1: icmp_seq=29 ttl=64 time=7747 ms
64 bytes from 10.0.0.1: icmp_seq=30 ttl=64 time=6739 ms
64 bytes from 10.0.0.1: icmp_seq=31 ttl=64 time=5721 ms
64 bytes from 10.0.0.1: icmp_seq=32 ttl=64 time=4722 ms
64 bytes from 10.0.0.1: icmp_seq=33 ttl=64 time=3721 ms
64 bytes from 10.0.0.1: icmp_seq=34 ttl=64 time=2720 ms
64 bytes from 10.0.0.1: icmp_seq=35 ttl=64 time=1719 ms
64 bytes from 10.0.0.1: icmp_seq=36 ttl=64 time=719 ms
64 bytes from 10.0.0.1: icmp_seq=37 ttl=64 time=63.7 ms
64 bytes from 10.0.0.1: icmp_seq=38 ttl=64 time=64.2 ms
64 bytes from 10.0.0.1: icmp_seq=39 ttl=64 time=72.9 ms
64 bytes from 10.0.0.1: icmp_seq=40 ttl=64 time=72.1 ms
64 bytes from 10.0.0.1: icmp_seq=41 ttl=64 time=71.1 ms
64 bytes from 10.0.0.1: icmp_seq=42 ttl=64 time=94.9 ms
64 bytes from 10.0.0.1: icmp_seq=43 ttl=64 time=67.3 ms
^C
--- 10.0.0.1 ping statistics ---
43 packets transmitted, 43 received, 0% packet loss, time 42591ms
rtt min/avg/max/mdev = 63.736/3633.659/8511.203/3435.161 ms, pipe 9

it turns out that due to high ping (latency) the speed decreases

M0dEx commented 4 months ago

Does the latency increase only when going through the Quincy tunnel, or does it spike like this even for traffic outside of it?

vukitoso commented 4 months ago

only when going through the Quincy tunnel

M0dEx commented 4 months ago

Could you try the same thing with WireGuard? It does seem like an issue on Quincy's part, but if WireGuard behaves similarly, it might just be the hosting provider throttling UDP traffic.

vukitoso commented 4 months ago

I carried out several tests, the results are very interesting. Let me start with the fact that the single-board armv7 with which I did the first tests is located at my home. At home I have LTE internet. Speed ​​up to approximately 50 Mbps.

There is also a VDS (Debian, x86_64) on the Internet. My Internet provider unfortunately blocks the WireGuard connection, so I tested with WireGuard from “VDS 100Mbps”.

Let me remind you that I have:

"VPN Server"
KVM VDS x86_64:
Debian 12
CPU: Limit 30% from 2300 MHz (=690 MHz)
RAM: 512 Mb
eth0: mtu 1500
tun0: mtu 1400
Internet speed is limited to exactly 10 Mbit/s
Quincy 0.6.1 binary from github - quincy-linux-x86_64.tar.gz
ip: 111.111.111.111

"VDS 100Mbps"
KVM VDS x86_64
Debian 12
CPU: 2700 MHz
RAM: 512 Mb
eth0: mtu 1500
tun0: mtu 1400
Internet speed: 100 Mbit/s
Quincy 0.6.1 binary from github - quincy-linux-x86_64.tar.gz

My local nodes:

"Odroid c1"
ARMv7 board
Debian 12
CPU: 4 x 1500 MHz
RAM: 1024 Mb
eth0: mtu 1500
tun0: mtu 1400
LTE: Internet speed up to approximately 50 Mbit/s
Quincy 0.6.1 install from: "cargo install quincy"

"Bhyve FreeBSD"
Bhyve virtual on FreeBSD 13.2, x86_64
Debian 11
CPU: 3500 MHz
RAM: 512 Mb
eth0: mtu 1500
tun0: mtu 1400
LTE: Internet speed up to approximately 50 Mbit/s
Quincy 0.6.1 binary from github - quincy-linux-x86_64.tar.gz

The Quincy server runs on a "VPN Server", where the Internet speed is limited to 10 Mbps. IP address of the "VPN Server" server was replaced with 111.111.111.111

In short, the test result is as follows: High ping of up to 12 seconds is reproduced only on "Odroid c1" and the speed is greatly reduced, to 1.3 Mbit/s. From my computer with the Bhyve FreeBSD virtual machine, the maximum ping is 1787ms. But the speed is already higher, up to 4.6 Mbit/s. At high-speed "VDS 100Mbps" the speed is very close to the maximum possible - 9.7 Mbit/s, ping up to 267ms.

I conducted the WireGuard test: "VDS 100Mbps" -> "VPN Server". Result: ping is normal, for tcp test 30-179ms, for udp test ping unchanged: 29ms

More details below, each test in a separate message.

vukitoso commented 4 months ago 
WireGuard
"VDS 100Mbps" -> "VPN Server".

Normal ping without load
ping 10.0.0.1
ping: 29 - 33 (min - max)
29 packets transmitted, 29 received, 0% packet loss, time 28041ms
rtt min/avg/max/mdev = 29.461/30.265/33.590/1.023 ms

TCP

iperf3 -c 111.111.111.111 -P1 --bitrate 5M
[  5]   0.00-10.00  sec  6.00 MBytes  5.03 Mbits/sec    4             sender
[  5]   0.00-10.03  sec  6.00 MBytes  5.02 Mbits/sec                  receiver
ping 29 - 121
18 packets transmitted, 18 received, 0% packet loss, time 17021ms
rtt min/avg/max/mdev = 29.400/45.075/121.708/27.582 ms

iperf3 -c 111.111.111.111 -P1 --bitrate 9M
[  5]   0.00-10.00  sec  10.8 MBytes  9.02 Mbits/sec    2             sender
[  5]   0.00-10.04  sec  10.8 MBytes  8.98 Mbits/sec                  receiver
ping 29 - 127
16 packets transmitted, 16 received, 0% packet loss, time 15019ms
rtt min/avg/max/mdev = 29.587/60.531/127.393/33.570 ms

iperf3 -c 111.111.111.111 -P1 --bitrate 10M
[  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec    0             sender
[  5]   0.00-10.20  sec  12.0 MBytes  9.87 Mbits/sec                  receiver
ping 29 - 179
18 packets transmitted, 18 received, 0% packet loss, time 17021ms
rtt min/avg/max/mdev = 29.556/87.551/179.142/57.020 ms

UDP
iperf3 -c 111.111.111.111 -P1 -u --bitrate 10M
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/8632 (0%)  sender
[  5]   0.00-10.03  sec  11.9 MBytes  9.97 Mbits/sec  0.372 ms  0/8632 (0%)  receiver
ping 29
17 packets transmitted, 17 received, 0% packet loss, time 16021ms
rtt min/avg/max/mdev = 29.396/30.075/31.480/0.638 ms

iperf3 -c 111.111.111.111 -P1 -u --bitrate 12M
[  5]   0.00-10.00  sec  14.3 MBytes  12.0 Mbits/sec  0.000 ms  0/10359 (0%)  sender
[  5]   0.00-11.10  sec  13.4 MBytes  10.1 Mbits/sec  0.437 ms  673/10359 (6.5%)  receiver
ping 29 - 1087
18 packets transmitted, 16 received, 11.1111% packet loss, time 17036ms
rtt min/avg/max/mdev = 29.540/464.743/1087.792/428.997 ms, pipe 2

iperf3 -c 111.111.111.111 -P1 -u --bitrate 14M
[  5]   0.00-10.00  sec  16.7 MBytes  14.0 Mbits/sec  0.000 ms  0/12085 (0%)  sender
[  5]   0.00-11.12  sec  13.4 MBytes  10.1 Mbits/sec  0.288 ms  2390/12085 (20%)  receiver
ping 29 - 1131
20 packets transmitted, 18 received, 10% packet loss, time 19068ms
rtt min/avg/max/mdev = 29.478/458.143/1131.452/491.420 ms, pipe 2

iperf3 -c 111.111.111.111 -P1 -u --bitrate 16M
[  5]   0.00-10.00  sec  19.1 MBytes  16.0 Mbits/sec  0.000 ms  0/13811 (0%)  sender
[  5]   0.00-11.12  sec  13.4 MBytes  10.1 Mbits/sec  0.323 ms  4108/13811 (30%)  receiver
ping 29 - 1103
18 packets transmitted, 15 received, 16.6667% packet loss, time 17102ms
rtt min/avg/max/mdev = 29.566/489.902/1103.806/484.439 ms, pipe 2

==========================

TCP

iperf3 -c 10.0.0.1 -P1 --bitrate 5M
[  5]   0.00-10.00  sec  6.00 MBytes  5.03 Mbits/sec    0             sender
[  5]   0.00-10.03  sec  6.00 MBytes  5.02 Mbits/sec                  receiver
ping 29 - 112
20 packets transmitted, 20 received, 0% packet loss, time 19024ms
rtt min/avg/max/mdev = 29.548/40.899/112.150/22.234 ms

iperf3 -c 10.0.0.1 -P1 --bitrate 10M
[  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec    0             sender
[  5]   0.00-10.58  sec  12.0 MBytes  9.48 Mbits/sec                  receiver
ping 29 - 536
20 packets transmitted, 20 received, 0% packet loss, time 19024ms
rtt min/avg/max/mdev = 29.533/183.049/536.245/179.372 ms

iperf3 -c 10.0.0.1 -P1 --bitrate 12M
[  5]   0.00-10.00  sec  12.5 MBytes  10.5 Mbits/sec    0             sender
[  5]   0.00-10.58  sec  11.9 MBytes  9.46 Mbits/sec                  receiver
ping 29 - 580
20 packets transmitted, 20 received, 0% packet loss, time 19024ms
rtt min/avg/max/mdev = 29.562/195.268/580.462/190.267 ms

iperf3 -c 10.0.0.1 -P10
[SUM]   0.00-10.00  sec  15.2 MBytes  12.7 Mbits/sec  407             sender
[SUM]   0.00-11.11  sec  12.7 MBytes  9.56 Mbits/sec                  receiver
ping 29 - 1136
20 packets transmitted, 19 received, 5% packet loss, time 19049ms
rtt min/avg/max/mdev = 29.570/426.168/1136.811/416.748 ms, pipe 2

UDP

iperf3 -c 10.0.0.1 -P1 -u --bitrate 5M
[  5]   0.00-10.00  sec  5.96 MBytes  5.00 Mbits/sec  0.000 ms  0/4569 (0%)  sender
[  5]   0.00-10.03  sec  5.96 MBytes  4.98 Mbits/sec  0.147 ms  1/4569 (0.022%)  receiver
ping 29 - 60
20 packets transmitted, 20 received, 0% packet loss, time 19023ms
rtt min/avg/max/mdev = 29.234/31.620/60.917/6.883 ms

iperf3 -c 10.0.0.1 -P1 -u --bitrate 8M
[  5]   0.00-10.00  sec  9.54 MBytes  8.00 Mbits/sec  0.000 ms  0/7310 (0%)  sender
[  5]   0.00-10.03  sec  9.54 MBytes  7.98 Mbits/sec  0.052 ms  0/7310 (0%)  receiver
ping 29 - 31
20 packets transmitted, 20 received, 0% packet loss, time 19023ms
rtt min/avg/max/mdev = 29.258/30.065/31.853/0.666 ms

iperf3 -c 10.0.0.1 -P1 -u --bitrate 10M
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/9137 (0%)  sender
[  5]   0.00-10.43  sec  11.9 MBytes  9.58 Mbits/sec  0.279 ms  0/9137 (0%)  receiver
ping 29 - 417
20 packets transmitted, 20 received, 0% packet loss, time 19017ms
rtt min/avg/max/mdev = 29.533/135.717/417.658/131.109 ms

iperf3 -c 10.0.0.1 -P1 -u --bitrate 12M
[  5]   0.00-10.00  sec  14.3 MBytes  12.0 Mbits/sec  0.000 ms  0/10964 (0%)  sender
[  5]   0.00-11.10  sec  12.7 MBytes  9.60 Mbits/sec  0.629 ms  1228/10964 (11%)  receiver
ping 29 - 1097
20 packets transmitted, 17 received, 15% packet loss, time 19078ms
rtt min/avg/max/mdev = 29.410/363.365/1097.805/428.200 ms, pipe 2

iperf3 -c 10.0.0.1 -P1 -u --bitrate 14M
[  5]   0.00-10.00  sec  16.7 MBytes  14.0 Mbits/sec  0.000 ms  0/12792 (0%)  sender
[  5]   0.00-11.11  sec  12.7 MBytes  9.62 Mbits/sec  0.345 ms  3030/12791 (24%)  receiver
ping 29 - 1121
20 packets transmitted, 17 received, 15% packet loss, time 19088ms
rtt min/avg/max/mdev = 29.599/429.196/1121.303/474.560 ms, pipe 2
vukitoso commented 4 months ago 
Quincy
"VDS 100Mbps" -> "VPN Server"

ping -c 20 10.0.0.1
ping 30 - 33
20 packets transmitted, 20 received, 0% packet loss, time 19030ms
rtt min/avg/max/mdev = 30.037/30.768/33.393/0.780 ms

TCP

iperf3 -c 10.0.0.1 -P1 --bitrate 5M
[  5]   0.00-10.00  sec  6.00 MBytes  5.03 Mbits/sec    0             sender
[  5]   0.00-10.03  sec  6.00 MBytes  5.02 Mbits/sec                  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19026ms
rtt min/avg/max/mdev = 30.064/52.912/202.071/43.979 ms
ping 30-202

iperf3 -c 10.0.0.1 -P1 --bitrate 8M
[  5]   0.00-10.00  sec  9.62 MBytes  8.07 Mbits/sec    0             sender
[  5]   0.00-10.10  sec  9.62 MBytes  7.99 Mbits/sec                  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19023ms
rtt min/avg/max/mdev = 29.968/58.000/147.554/35.649 ms
ping 29 - 147

iperf3 -c 10.0.0.1 -P1 --bitrate 10M
[  5]   0.00-10.00  sec  12.0 MBytes  10.1 Mbits/sec    0             sender
[  5]   0.00-10.61  sec  11.7 MBytes  9.29 Mbits/sec                  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19021ms
rtt min/avg/max/mdev = 29.980/199.092/525.697/182.688 ms
ping 29 - 525

iperf3 -c 10.0.0.1 -P1 --bitrate 12M
[  5]   0.00-10.00  sec  12.1 MBytes  10.2 Mbits/sec    0             sender
[  5]   0.00-10.58  sec  11.6 MBytes  9.19 Mbits/sec                  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19018ms
rtt min/avg/max/mdev = 30.059/211.939/585.029/198.619 ms
ping 30 - 585

iperf3 -c 10.0.0.1 -P2
[SUM]   0.00-10.00  sec  12.9 MBytes  10.9 Mbits/sec    0             sender
[SUM]   0.00-10.65  sec  11.8 MBytes  9.28 Mbits/sec                  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19025ms
rtt min/avg/max/mdev = 29.853/232.650/1012.833/259.596 ms
ping 29 - 1012

iperf3 -c 10.0.0.1 -P3
quincy-client crashed
2024-02-14T11:00:23.115816Z ERROR quincy_client: A critical error occurred: datagram too large

iperf3 -c 10.0.0.1 -P10
quincy-client crashed
2024-02-14T10:48:35.047540Z ERROR quincy_client: A critical error occurred: datagram too large

iperf3 -c 10.0.0.1 -P10
[SUM]   0.00-10.00  sec  15.1 MBytes  12.6 Mbits/sec  232             sender
[SUM]   0.00-11.04  sec  12.2 MBytes  9.27 Mbits/sec                  receiver
20 packets transmitted, 19 received, 5% packet loss, time 19072ms
rtt min/avg/max/mdev = 29.875/468.474/1398.637/469.510 ms, pipe 2
ping 29 - 1398

UDP

iperf3 -c 10.0.0.1 -P1 -u --bitrate 5M
[  5]   0.00-10.00  sec  5.96 MBytes  5.00 Mbits/sec  0.000 ms  0/4637 (0%)  sender
[  5]   0.00-10.03  sec  5.96 MBytes  4.99 Mbits/sec  0.091 ms  0/4637 (0%)  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19023ms
rtt min/avg/max/mdev = 29.591/30.488/33.201/1.037 ms
ping 29 - 33

iperf3 -c 10.0.0.1 -P1 -u --bitrate 8M
[  5]   0.00-10.00  sec  9.54 MBytes  8.00 Mbits/sec  0.000 ms  0/7418 (0%)  sender
[  5]   0.00-10.04  sec  9.54 MBytes  7.97 Mbits/sec  1.722 ms  0/7418 (0%)  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19004ms
rtt min/avg/max/mdev = 29.618/33.406/47.496/5.920 ms
ping 29 - 47

iperf3 -c 10.0.0.1 -P1 -u --bitrate 10M
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/9273 (0%)  sender
[  5]   0.00-10.30  sec  11.9 MBytes  9.71 Mbits/sec  1.291 ms  0/9273 (0%)  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19024ms
rtt min/avg/max/mdev = 30.098/100.390/267.750/83.541 ms
ping 30 - 267

iperf3 -c 10.0.0.1 -P1 -u --bitrate 11M
quincy-client crashed
2024-02-14T10:56:35.674042Z ERROR quincy_client: A critical error occurred: datagram too large

iperf3 -c 10.0.0.1 -P1 -u --bitrate 12M
quincy-client crashed
2024-02-14T10:53:37.975557Z ERROR quincy_client: A critical error occurred: datagram too large

iperf3 -c 10.0.0.1 -P1 -u --bitrate 12M
quincy-client crashed
2024-02-14T10:54:51.914116Z ERROR quincy_client: A critical error occurred: datagram too large

iperf3 -c 10.0.0.1 -P3 -u --bitrate 4M
[SUM]   0.00-10.00  sec  14.3 MBytes  12.0 Mbits/sec  0.000 ms  0/11127 (0%)  sender
[SUM]   0.00-11.67  sec  13.2 MBytes  9.53 Mbits/sec  1.792 ms  821/11127 (7.4%)  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19034ms
rtt min/avg/max/mdev = 29.740/598.867/1656.669/595.277 ms, pipe 2
ping 29 - 1656

iperf3 -c 10.0.0.1 -P3 -u --bitrate 5M
quincy-client crashed
vukitoso commented 4 months ago 
Quincy
"Odroid c1" -> "VPN Server"

Normal ping without load
ping -c 20 10.0.0.1
20 packets transmitted, 20 received, 0% packet loss, time 19020ms
rtt min/avg/max/mdev = 112.185/119.757/136.319/6.691 ms
ping 112 - 136

TCP

iperf3 -c 10.0.0.1 -P1 --bitrate 10M
[  5]   0.00-10.00  sec  2.00 MBytes  1.68 Mbits/sec    2             sender
[  5]   0.00-11.75  sec  1.76 MBytes  1.26 Mbits/sec                  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19020ms
rtt min/avg/max/mdev = 104.915/828.171/1741.896/617.205 ms, pipe 2
ping 104 - 1741

iperf3 -c 10.0.0.1 -P1 --bitrate 5M
[  5]   0.00-10.00  sec  1.50 MBytes  1.26 Mbits/sec    9             sender
[  5]   0.00-11.18  sec  1.30 MBytes   978 Kbits/sec                  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19121ms
rtt min/avg/max/mdev = 105.962/545.795/1266.248/432.891 ms, pipe 2
ping 105 - 1266

UDP

iperf3 -c 10.0.0.1 -P1 -u --bitrate 10M
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/9273 (0%)  sender
[  5]   0.00-24.30  sec  1.78 MBytes   614 Kbits/sec  8.621 ms  7889/9273 (85%)  receiver
20 packets transmitted, 3 received, 85% packet loss, time 19614ms
rtt min/avg/max/mdev = 108.650/110.324/111.768/1.283 ms
fail

[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/9273 (0%)  sender
[  5]   0.00-18.46  sec  2.37 MBytes  1.08 Mbits/sec  6.593 ms  7429/9273 (80%)  receiver
20 packets transmitted, 12 received, 40% packet loss, time 19528ms
rtt min/avg/max/mdev = 116.527/3237.037/7936.438/2625.055 ms, pipe 7
ping 116 - 7936

iperf3 -c 10.0.0.1 -P1 -u --bitrate 9M
[  5]   0.00-10.00  sec  10.7 MBytes  9.00 Mbits/sec  0.000 ms  0/8346 (0%)  sender
[  5]   0.00-22.51  sec  1.82 MBytes   678 Kbits/sec  14.232 ms  6930/8346 (83%)  receiver
20 packets transmitted, 12 received, 40% packet loss, time 19671ms
rtt min/avg/max/mdev = 123.623/6176.599/12049.691/3931.079 ms, pipe 8
ping 123 - 12049

iperf3 -c 10.0.0.1 -P1 -u --bitrate 5M
[  5]   0.00-10.00  sec  5.96 MBytes  5.00 Mbits/sec  0.000 ms  0/4637 (0%)  sender
[  5]   0.00-17.61  sec  2.66 MBytes  1.27 Mbits/sec  9.355 ms  2566/4637 (55%)  receiver
20 packets transmitted, 14 received, 30% packet loss, time 19487ms
rtt min/avg/max/mdev = 115.932/2746.392/7598.410/2488.151 ms, pipe 8
ping 115 - 7598
vukitoso commented 4 months ago 
Quincy
"Bhyve FreeBSD" -> "VPN Server"

Normal ping without load
ping -c 20 10.0.0.1
20 packets transmitted, 20 received, 0% packet loss, time 19156ms
rtt min/avg/max/mdev = 107.338/118.714/161.888/11.812 ms
ping 107 - 161

TCP

iperf3 -c 10.0.0.1 -P1
2024-02-14T15:27:06.520335Z ERROR quinn_udp::imp: got transmit error, halting segmentation offload
2024-02-14T15:27:06.520351Z  WARN quinn_udp: sendmsg error: Os { code: 5, kind: Uncategorized, message: "Input/output error" }, Transmit: { destination: 111.111.111.111:55555, src_ip: None, enc: Some(Ect0), len: 11523, segment_size: Some(1442) }
2024-02-14T15:27:09.060359Z ERROR quincy_client: A critical error occurred: datagram too large

iperf3 -c 10.0.0.1 -P1 --bitrate 3M
[  5]   0.00-10.00  sec  3.68 MBytes  3.09 Mbits/sec    0             sender
[  5]   0.00-10.18  sec  3.68 MBytes  3.04 Mbits/sec                  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19101ms
rtt min/avg/max/mdev = 102.619/124.376/217.422/29.779 ms
ping 102 - 217

iperf3 -c 10.0.0.1 -P1 --bitrate 4M
2024-02-14T15:04:58.799506Z ERROR quinn_udp::imp: got transmit error, halting segmentation offload
2024-02-14T15:04:58.799523Z  WARN quinn_udp: sendmsg error: Os { code: 5, kind: Uncategorized, message: "Input/output error" }, Transmit: { destination: 111.111.111.111:55555, src_ip: None, enc: Some(Ect0), len: 11523, segment_size: Some(1442) }
2024-02-14T15:05:05.982892Z ERROR quincy_client: A critical error occurred: datagram too large

iperf3 -c 10.0.0.1 -P1 --bitrate 5M
[  5]   0.00-10.00  sec  2.63 MBytes  2.20 Mbits/sec    8             sender
[  5]   0.00-10.41  sec  2.47 MBytes  1.99 Mbits/sec                  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19039ms
rtt min/avg/max/mdev = 101.163/152.338/417.993/77.008 ms
ping 101 - 417

iperf3 -c 10.0.0.1 -P1 --bitrate 10M
2024-02-14T14:32:45.419913Z ERROR quinn_udp::imp: got transmit error, halting segmentation offload
2024-02-14T14:32:45.419929Z  WARN quinn_udp: sendmsg error: Os { code: 5, kind: Uncategorized, message: "Input/output error" }, Transmit: { destination: 111.111.111.111:55555, src_ip: None, enc: Some(Ect0), len: 11523, segment_size: Some(1442) }
2024-02-14T14:32:51.907596Z ERROR quincy_client: A critical error occurred: datagram too large

iperf3 -c 10.0.0.1 -P3 --bitrate 2M
2024-02-14T15:07:55.233064Z ERROR quinn_udp::imp: got transmit error, halting segmentation offload
2024-02-14T15:07:55.233081Z  WARN quinn_udp: sendmsg error: Os { code: 5, kind: Uncategorized, message: "Input/output error" }, Transmit: { destination: 111.111.111.111:55555, src_ip: None, enc: Some(Ect0), len: 11523, segment_size: Some(1442) }
2024-02-14T15:08:04.344004Z ERROR quincy_client: A critical error occurred: datagram too large

iperf3 -c 10.0.0.1 -P5 --bitrate 1M
2024-02-14T15:08:44.854864Z ERROR quinn_udp::imp: got transmit error, halting segmentation offload
2024-02-14T15:08:44.854881Z  WARN quinn_udp: sendmsg error: Os { code: 5, kind: Uncategorized, message: "Input/output error" }, Transmit: { destination: 111.111.111.111:55555, src_ip: None, enc: Some(Ect0), len: 11523, segment_size: Some(1442) }
2024-02-14T15:08:48.819320Z ERROR quincy_client: A critical error occurred: datagram too large

UDP

iperf3 -c 10.0.0.1 -P1 -u --bitrate 14M
[  5]   0.00-10.00  sec  16.7 MBytes  14.0 Mbits/sec  0.000 ms  0/12982 (0%)  sender
[  5]   0.00-11.06  sec  6.05 MBytes  4.59 Mbits/sec  2.018 ms  8276/12982 (64%)  receiver
20 packets transmitted, 13 received, 35% packet loss, time 19212ms
rtt min/avg/max/mdev = 103.110/284.551/939.276/290.333 ms
ping 103 - 939

iperf3 -c 10.0.0.1 -P1 -u --bitrate 12M
[  5]   0.00-10.00  sec  14.3 MBytes  12.0 Mbits/sec  0.000 ms  0/11127 (0%)  sender
[  5]   0.00-11.02  sec  5.72 MBytes  4.35 Mbits/sec  0.293 ms  6677/11127 (60%)  receiver
20 packets transmitted, 13 received, 35% packet loss, time 19183ms
rtt min/avg/max/mdev = 111.792/338.670/865.731/328.196 ms
ping 111 - 865

iperf3 -c 10.0.0.1 -P1 -u --bitrate 10M
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/9273 (0%)  sender
[  5]   0.00-11.39  sec  4.35 MBytes  3.21 Mbits/sec  0.604 ms  5887/9273 (63%)  receiver
20 packets transmitted, 14 received, 30% packet loss, time 19177ms
rtt min/avg/max/mdev = 99.325/417.286/1154.542/422.921 ms, pipe 2
ping 99 - 1154

iperf3 -c 10.0.0.1 -P1 -u --bitrate 9M
[  5]   0.00-10.00  sec  10.7 MBytes  9.00 Mbits/sec  0.000 ms  0/8345 (0%)  sender
[  5]   0.00-10.87  sec  5.95 MBytes  4.59 Mbits/sec  1.224 ms  3714/8345 (45%)  receiver
20 packets transmitted, 16 received, 20% packet loss, time 19139ms
rtt min/avg/max/mdev = 101.978/438.091/1042.838/425.681 ms, pipe 2
ping 101 - 1042

iperf3 -c 10.0.0.1 -P1 -u --bitrate 5M
[  5]   0.00-10.00  sec  5.96 MBytes  5.00 Mbits/sec  0.000 ms  0/4637 (0%)  sender
[  5]   0.00-10.12  sec  4.94 MBytes  4.09 Mbits/sec  2.009 ms  796/4637 (17%)  receiver
20 packets transmitted, 17 received, 15% packet loss, time 19133ms
rtt min/avg/max/mdev = 100.775/560.447/1787.948/641.815 ms, pipe 2
ping 100 - 1787

iperf3 -c 10.0.0.1 -P1 -u --bitrate 3M
[  5]   0.00-10.00  sec  3.58 MBytes  3.00 Mbits/sec  0.000 ms  0/2782 (0%)  sender
[  5]   0.00-10.12  sec  3.57 MBytes  2.96 Mbits/sec  3.949 ms  4/2782 (0.14%)  receiver
20 packets transmitted, 20 received, 0% packet loss, time 19061ms
rtt min/avg/max/mdev = 101.331/568.254/1703.521/620.751 ms, pipe 2
ping 101 - 1703
M0dEx commented 4 months ago

Could you try lowering the MTU in the config? Maybe 1300 B?

vukitoso commented 4 months ago

Quincy "Bhyve FreeBSD" -> "VPN Server" MTU 1300

error

$ iperf3 -c 10.0.0.1 -P1 --bitrate 5M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 57872 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   366 KBytes  3.00 Mbits/sec    0   36.0 Bytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes  
...

client error

$ ./quincy-client --config-path /opt/quincy/examples/client.toml
2024-02-14T17:53:38.546721Z  INFO quincy::client: Connecting: 111.111.111.111:55555
2024-02-14T17:53:38.546768Z  WARN quincy::utils::socket: Unable to set desired send buffer size. Desired: 2097152, Actual: 425984
2024-02-14T17:53:38.546787Z  WARN quincy::utils::socket: Unable to set desired recv buffer size. Desired: 2097152, Actual: 425984
2024-02-14T17:53:38.671841Z  INFO quincy::client: Connection established: 111.111.111.111:55555
2024-02-14T17:53:39.256488Z  INFO quincy::client: Successfully authenticated
2024-02-14T17:53:39.256539Z  INFO quincy::client: Received client address: 10.0.0.2/24
2024-02-14T17:54:34.175117Z ERROR quinn_udp::imp: got transmit error, halting segmentation offload
2024-02-14T17:54:34.175146Z  WARN quinn_udp: sendmsg error: Os { code: 5, kind: Uncategorized, message: "Input/output error" }, Transmit: { destination: 111.111.111.111:55555, src_ip: None, enc: Some(Ect0), len: 10723, segment_size: Some(1342) }
2024-02-14T17:55:18.474967Z ERROR quincy_client: A critical error occurred: datagram too large
M0dEx commented 4 months ago

I just now noticed, you are running one of the benchmarks on FreeBSD? *BSD is not a supported platform, with the reason for a lack of support being the TUN library, so I am quite surprised you managed to get it to run at all.

vukitoso commented 4 months ago

I just now noticed, you are running one of the benchmarks on FreeBSD?

No. This is virual Linux Debian 11 running on bhyve as a guest OS.

$ uname -a
Linux debian11 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux

$ ./quincy-client --config-path /opt/quincy/examples/client.toml
2024-02-14T20:14:52.445777Z  INFO quincy::client: Connecting: 111.111.111.111:55555
2024-02-14T20:14:52.445936Z  WARN quincy::utils::socket: Unable to set desired send buffer size. Desired: 2097152, Actual: 425984
2024-02-14T20:14:52.446016Z  WARN quincy::utils::socket: Unable to set desired recv buffer size. Desired: 2097152, Actual: 425984
2024-02-14T20:14:52.570313Z  INFO quincy::client: Connection established: 111.111.111.111:55555
2024-02-14T20:14:53.194021Z  INFO quincy::client: Successfully authenticated
2024-02-14T20:14:53.194150Z  INFO quincy::client: Received client address: 10.0.0.2/24

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 58:9c:fc:0d:27:48 brd ff:ff:ff:ff:ff:ff
    inet 10.10.100.3/24 brd 10.10.100.255 scope global enp0s5
       valid_lft forever preferred_lft forever
    inet6 fe80::5a9c:fcff:fe0d:2748/64 scope link 
       valid_lft forever preferred_lft forever
4: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1300 qdisc pfifo_fast state UNKNOWN group default qlen 500
    link/none 
    inet 10.0.0.2 peer 10.0.0.0/24 scope global tun0
       valid_lft forever preferred_lft forever
    inet6 fe80::d15c:d094:ddf9:7dc/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
vukitoso commented 4 months ago

It worked N times. On my LTE connection it’s like this, the speed decreases, I don’t know why, maybe the provider is making adjustments, maybe something else.

$ iperf3 -c 10.0.0.1 -P1 --bitrate 5M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 39472 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   101 KBytes   829 Kbits/sec    8   11.0 KBytes       
[  5]   1.00-2.00   sec  91.4 KBytes   749 Kbits/sec    0   14.6 KBytes       
[  5]   2.00-3.00   sec   165 KBytes  1.35 Mbits/sec    0   20.7 KBytes       
[  5]   3.00-4.00   sec   182 KBytes  1.49 Mbits/sec    0   25.6 KBytes       
[  5]   4.00-5.00   sec   286 KBytes  2.35 Mbits/sec    0   31.7 KBytes       
[  5]   5.00-6.00   sec   321 KBytes  2.63 Mbits/sec    0   36.6 KBytes       
[  5]   6.00-7.00   sec   306 KBytes  2.51 Mbits/sec    0   53.6 KBytes       
[  5]   7.00-8.00   sec   461 KBytes  3.77 Mbits/sec    0   87.8 KBytes       
[  5]   8.00-9.00   sec   512 KBytes  4.19 Mbits/sec    0    134 KBytes       
[  5]   9.00-10.00  sec   512 KBytes  4.19 Mbits/sec    0    196 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.87 MBytes  2.41 Mbits/sec    8             sender
[  5]   0.00-10.49  sec  2.64 MBytes  2.11 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 5M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 49418 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   102 KBytes   838 Kbits/sec    8   11.0 KBytes       
[  5]   1.00-2.00   sec  75.6 KBytes   619 Kbits/sec    0   14.6 KBytes       
[  5]   2.00-3.00   sec   136 KBytes  1.12 Mbits/sec    0   19.5 KBytes       
[  5]   3.00-4.00   sec   243 KBytes  1.99 Mbits/sec    0   25.6 KBytes       
[  5]   4.00-5.00   sec   216 KBytes  1.77 Mbits/sec    0   30.5 KBytes       
[  5]   5.00-6.00   sec   289 KBytes  2.37 Mbits/sec    0   35.3 KBytes       
[  5]   6.00-7.00   sec   328 KBytes  2.69 Mbits/sec    0   53.6 KBytes       
[  5]   7.00-8.00   sec   458 KBytes  3.75 Mbits/sec    0   87.8 KBytes       
[  5]   8.00-9.00   sec   639 KBytes  5.23 Mbits/sec    0    134 KBytes       
[  5]   9.00-10.00  sec   640 KBytes  5.24 Mbits/sec    0    197 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.05 MBytes  2.56 Mbits/sec    8             sender
[  5]   0.00-10.35  sec  2.81 MBytes  2.28 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 7M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.3 port 60466 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   104 KBytes   848 Kbits/sec    8   11.0 KBytes       
[  5]   1.00-2.00   sec   115 KBytes   938 Kbits/sec    0   15.8 KBytes       
[  5]   2.00-3.00   sec   136 KBytes  1.12 Mbits/sec    0   20.7 KBytes       
[  5]   3.00-4.00   sec   244 KBytes  2.00 Mbits/sec    0   26.8 KBytes       
[  5]   4.00-5.00   sec   292 KBytes  2.40 Mbits/sec    0   31.7 KBytes       
[  5]   5.00-6.00   sec   286 KBytes  2.35 Mbits/sec    0   37.8 KBytes       
[  5]   6.00-7.00   sec   413 KBytes  3.38 Mbits/sec    0   56.1 KBytes       
[  5]   7.00-8.00   sec   458 KBytes  3.75 Mbits/sec    0   87.8 KBytes       
[  5]   8.00-9.00   sec   512 KBytes  4.19 Mbits/sec    0    135 KBytes       
[  5]   9.00-10.00  sec   512 KBytes  4.19 Mbits/sec    0    196 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.00 MBytes  2.52 Mbits/sec    8             sender
[  5]   0.00-10.39  sec  2.79 MBytes  2.26 Mbits/sec                  receiver

On bhyve with UDP it is better than on odroid.

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 10M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 56441 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  1.19 MBytes  9.99 Mbits/sec  1001  
[  5]   1.00-2.00   sec  1.19 MBytes  10.0 Mbits/sec  1002  
[  5]   2.00-3.00   sec  1.19 MBytes  10.0 Mbits/sec  1002  
[  5]   3.00-4.00   sec  1.19 MBytes  9.99 Mbits/sec  1001  
[  5]   4.00-5.00   sec  1.19 MBytes  10.0 Mbits/sec  1002  
[  5]   5.00-6.00   sec  1.19 MBytes  9.99 Mbits/sec  1001  
[  5]   6.00-7.00   sec  1.19 MBytes  10.0 Mbits/sec  1002  
[  5]   7.00-8.00   sec  1.19 MBytes  10.0 Mbits/sec  1002  
[  5]   8.00-9.00   sec  1.19 MBytes  9.99 Mbits/sec  1001  
[  5]   9.00-10.00  sec  1.19 MBytes  10.0 Mbits/sec  1002  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/10016 (0%)  sender
[  5]   0.00-11.01  sec  5.53 MBytes  4.21 Mbits/sec  0.361 ms  5372/10016 (54%)  receiver

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 8M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 50692 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   976 KBytes  8.00 Mbits/sec  801  
[  5]   1.00-2.00   sec   976 KBytes  8.00 Mbits/sec  801  
[  5]   2.00-3.00   sec   977 KBytes  8.01 Mbits/sec  802  
[  5]   3.00-4.00   sec   976 KBytes  8.00 Mbits/sec  801  
[  5]   4.00-5.00   sec   976 KBytes  8.00 Mbits/sec  801  
[  5]   5.00-6.00   sec   977 KBytes  8.01 Mbits/sec  802  
[  5]   6.00-7.00   sec   976 KBytes  8.00 Mbits/sec  801  
[  5]   7.00-8.00   sec   976 KBytes  8.00 Mbits/sec  801  
[  5]   8.00-9.00   sec   976 KBytes  8.00 Mbits/sec  801  
[  5]   9.00-10.00  sec   977 KBytes  8.01 Mbits/sec  802  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  9.54 MBytes  8.00 Mbits/sec  0.000 ms  0/8013 (0%)  sender
[  5]   0.00-10.12  sec  9.54 MBytes  7.91 Mbits/sec  0.577 ms  0/8013 (0%)  receiver
M0dEx commented 4 months ago

Can you run the benchmark for a bit longer? It seems the throughput could potentially be better; it just takes time to ramp up.

As for the datagram too large errors, it seems that sometimes, the QUIC header is larger than was expected.

M0dEx commented 4 months ago

Could you try installing Quincy on all devices using the following command? It should contain a fix for the datagram too large errors.

$ cargo install quincy --git https://github.com/M0dEx/quincy.git --branch '#40/mtu-issue'
vukitoso commented 4 months ago

Could you try installing Quincy on all devices using the following command?

I installed it from binaries. Unfortunately, the client also crashes with an error.

$ ./quincy-client --config-path /opt/quincy/examples/client.toml
2024-02-14T20:42:41.865552Z  INFO quincy::client: Connecting: 111.111.111.111:55555
2024-02-14T20:42:41.866110Z  WARN quincy::utils::socket: Unable to set desired send buffer size. Desired: 2097152, Actual: 425984
2024-02-14T20:42:41.866406Z  WARN quincy::utils::socket: Unable to set desired recv buffer size. Desired: 2097152, Actual: 425984
2024-02-14T20:42:41.996861Z  INFO quincy::client: Connection established: 111.111.111.111:55555
2024-02-14T20:42:42.672455Z  INFO quincy::client: Successfully authenticated
2024-02-14T20:42:42.672517Z  INFO quincy::client: Received client address: 10.0.0.3/24
2024-02-14T20:42:55.732362Z ERROR quinn_udp::imp: got transmit error, halting segmentation offload
2024-02-14T20:42:55.732659Z  WARN quinn_udp: sendmsg error: Os { code: 5, kind: Uncategorized, message: "Input/output error" }, Transmit: { destination: 111.111.111.111:55555, src_ip: None, enc: Some(Ect0), len: 10723, segment_size: Some(1342) }
2024-02-14T20:43:03.209510Z ERROR quincy_client: A critical error occurred: datagram too large
vukitoso commented 4 months ago

Can you run the benchmark for a bit longer? It seems the throughput could potentially be better; it just takes time to ramp up.

Yes you are right. It accelerates over time. But unfortunately, very often the client crashes.

$ iperf3 -c 10.0.0.1 -P1 -t 20 --bitrate 7M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.3 port 57920 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   669 KBytes  5.48 Mbits/sec    0    105 KBytes       
[  5]   1.00-2.00   sec   768 KBytes  6.29 Mbits/sec    0    136 KBytes       
[  5]   2.00-3.00   sec   768 KBytes  6.29 Mbits/sec    0    172 KBytes       
[  5]   3.00-4.00   sec   768 KBytes  6.29 Mbits/sec    0    210 KBytes       
[  5]   4.00-5.00   sec   896 KBytes  7.34 Mbits/sec    0    249 KBytes       
[  5]   5.00-6.00   sec   768 KBytes  6.29 Mbits/sec    0    286 KBytes       
[  5]   6.00-7.00   sec   896 KBytes  7.34 Mbits/sec    0    325 KBytes       
[  5]   7.00-8.00   sec   896 KBytes  7.34 Mbits/sec    0    367 KBytes       
[  5]   8.00-9.00   sec  1.00 MBytes  8.39 Mbits/sec    0    411 KBytes       
[  5]   9.00-10.00  sec  1.00 MBytes  8.39 Mbits/sec    0    477 KBytes       
[  5]  10.00-11.00  sec  1.00 MBytes  8.39 Mbits/sec    0    606 KBytes       
[  5]  11.00-12.00  sec   768 KBytes  6.29 Mbits/sec    0    757 KBytes       
[  5]  12.00-13.00  sec   896 KBytes  7.34 Mbits/sec    0    787 KBytes       
[  5]  13.00-14.00  sec   896 KBytes  7.34 Mbits/sec    0   36.0 Bytes       
[  5]  14.00-15.00  sec   256 KBytes  2.10 Mbits/sec    0   36.0 Bytes       
[  5]  15.00-16.00  sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes       
^C[  5]  16.00-16.45  sec  0.00 Bytes  0.00 bits/sec    0   36.0 Bytes
M0dEx commented 4 months ago

Could you try installing Quincy on all devices using the following command?

I installed it from binaries. Unfortunately, the client also crashes with an error.

$ ./quincy-client --config-path /opt/quincy/examples/client.toml
2024-02-14T20:42:41.865552Z  INFO quincy::client: Connecting: 111.111.111.111:55555
2024-02-14T20:42:41.866110Z  WARN quincy::utils::socket: Unable to set desired send buffer size. Desired: 2097152, Actual: 425984
2024-02-14T20:42:41.866406Z  WARN quincy::utils::socket: Unable to set desired recv buffer size. Desired: 2097152, Actual: 425984
2024-02-14T20:42:41.996861Z  INFO quincy::client: Connection established: 111.111.111.111:55555
2024-02-14T20:42:42.672455Z  INFO quincy::client: Successfully authenticated
2024-02-14T20:42:42.672517Z  INFO quincy::client: Received client address: 10.0.0.3/24
2024-02-14T20:42:55.732362Z ERROR quinn_udp::imp: got transmit error, halting segmentation offload
2024-02-14T20:42:55.732659Z  WARN quinn_udp: sendmsg error: Os { code: 5, kind: Uncategorized, message: "Input/output error" }, Transmit: { destination: 111.111.111.111:55555, src_ip: None, enc: Some(Ect0), len: 10723, segment_size: Some(1342) }
2024-02-14T20:43:03.209510Z ERROR quincy_client: A critical error occurred: datagram too large

Just to be clear, did you install Quincy using the command I provided? There is no official binary with the fix yet, so the command is the only way to install it. I do not want to release a new version without knowing it fixes your issue.

I could release a beta version for you, if you would prefer.

vukitoso commented 4 months ago

I rebuilt only on the "Odroid c1" client

cargo install quincy --git https://github.com/M0dEx/quincy.git --branch '#40/mtu-issue'

The client no longer falls. It helped.

M0dEx commented 4 months ago

Did the throughput improve at all? Are there any issues still remaining?

vukitoso commented 4 months ago

This time I launched the server on another VDS, where the throughput is 100Mbps. The speed dropped. mtu 1400

TCP

$ iperf3 -c 10.0.0.1 -P1 --bitrate 10M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 57852 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   571 KBytes  4.68 Mbits/sec    3   48.7 KBytes       
[  5]   1.00-2.00   sec   317 KBytes  2.60 Mbits/sec    0   52.7 KBytes       
[  5]   2.00-3.00   sec   250 KBytes  2.05 Mbits/sec    2   54.0 KBytes       
[  5]   3.00-4.00   sec   141 KBytes  1.16 Mbits/sec    0   55.3 KBytes       
[  5]   4.00-5.00   sec   214 KBytes  1.75 Mbits/sec    2   56.6 KBytes       
[  5]   5.00-6.00   sec   150 KBytes  1.23 Mbits/sec    3   57.9 KBytes       
[  5]   6.00-7.00   sec   359 KBytes  2.94 Mbits/sec    0   61.9 KBytes       
[  5]   7.00-8.00   sec   298 KBytes  2.44 Mbits/sec    0   64.5 KBytes       
[  5]   8.00-9.00   sec   342 KBytes  2.80 Mbits/sec    0   69.8 KBytes       
[  5]   9.00-10.00  sec   301 KBytes  2.46 Mbits/sec    0   86.9 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.88 MBytes  2.41 Mbits/sec   10             sender
[  5]   0.00-10.29  sec  2.79 MBytes  2.28 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 8M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 46136 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   486 KBytes  3.98 Mbits/sec    3   38.2 KBytes       
[  5]   1.00-2.00   sec   379 KBytes  3.11 Mbits/sec    2   42.1 KBytes       
[  5]   2.00-3.00   sec   254 KBytes  2.08 Mbits/sec    0   46.1 KBytes       
[  5]   3.00-4.00   sec   254 KBytes  2.08 Mbits/sec    0   50.0 KBytes       
[  5]   4.00-5.00   sec   254 KBytes  2.08 Mbits/sec    1   56.6 KBytes       
[  5]   5.00-6.00   sec   251 KBytes  2.06 Mbits/sec    1   57.9 KBytes       
[  5]   6.00-7.00   sec   169 KBytes  1.39 Mbits/sec    1   59.2 KBytes       
[  5]   7.00-8.00   sec   376 KBytes  3.08 Mbits/sec    0   61.9 KBytes       
[  5]   8.00-9.00   sec   234 KBytes  1.92 Mbits/sec    0   67.1 KBytes       
[  5]   9.00-10.00  sec   146 KBytes  1.20 Mbits/sec    0   80.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.74 MBytes  2.30 Mbits/sec    8             sender
[  5]   0.00-10.23  sec  2.67 MBytes  2.19 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 6M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 48880 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec    1   80.3 KBytes       
[  5]   1.00-2.00   sec   256 KBytes  2.10 Mbits/sec    4   80.3 KBytes       
[  5]   2.00-3.00   sec   240 KBytes  1.97 Mbits/sec    1   81.6 KBytes       
[  5]   3.00-4.00   sec   265 KBytes  2.17 Mbits/sec    1   82.9 KBytes       
[  5]   4.00-5.00   sec   263 KBytes  2.15 Mbits/sec    0   84.2 KBytes       
[  5]   5.00-6.00   sec   384 KBytes  3.15 Mbits/sec    0   92.1 KBytes       
[  5]   6.00-7.00   sec   256 KBytes  2.10 Mbits/sec    1   69.8 KBytes       
[  5]   7.00-8.00   sec   256 KBytes  2.10 Mbits/sec    0   85.6 KBytes       
[  5]   8.00-9.01   sec   256 KBytes  2.09 Mbits/sec    1   86.9 KBytes       
[  5]   9.01-10.00  sec   256 KBytes  2.11 Mbits/sec    0   89.5 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.88 MBytes  2.41 Mbits/sec    9             sender
[  5]   0.00-10.31  sec  2.69 MBytes  2.19 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 4M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 44136 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   488 KBytes  4.00 Mbits/sec    3   48.7 KBytes       
[  5]   1.00-2.00   sec   272 KBytes  2.23 Mbits/sec    2   51.3 KBytes       
[  5]   2.00-3.00   sec   251 KBytes  2.06 Mbits/sec    1   52.7 KBytes       
[  5]   3.00-4.00   sec   101 KBytes   830 Kbits/sec    0   55.3 KBytes       
[  5]   4.00-5.00   sec   153 KBytes  1.25 Mbits/sec    1   57.9 KBytes       
[  5]   5.00-6.00   sec   251 KBytes  2.06 Mbits/sec    0   60.6 KBytes       
[  5]   6.00-7.00   sec   249 KBytes  2.04 Mbits/sec    0   63.2 KBytes       
[  5]   7.00-8.00   sec   281 KBytes  2.31 Mbits/sec    0   71.1 KBytes       
[  5]   8.00-9.00   sec   256 KBytes  2.10 Mbits/sec    0   88.2 KBytes       
[  5]   9.00-10.00  sec   384 KBytes  3.15 Mbits/sec    0    115 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.62 MBytes  2.20 Mbits/sec    7             sender
[  5]   0.00-10.47  sec  2.49 MBytes  1.99 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 2M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 44124 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec    2   31.6 KBytes       
[  5]   1.00-2.00   sec   256 KBytes  2.10 Mbits/sec    0   35.5 KBytes       
[  5]   2.00-3.00   sec   256 KBytes  2.10 Mbits/sec    0   40.8 KBytes       
[  5]   3.00-4.00   sec   256 KBytes  2.10 Mbits/sec    1   43.4 KBytes       
[  5]   4.00-5.00   sec   256 KBytes  2.10 Mbits/sec    2   46.1 KBytes       
[  5]   5.00-6.00   sec   256 KBytes  2.10 Mbits/sec    2   48.7 KBytes       
[  5]   6.00-7.00   sec   242 KBytes  1.98 Mbits/sec    1   51.3 KBytes       
[  5]   7.00-8.00   sec   270 KBytes  2.21 Mbits/sec    0   55.3 KBytes       
[  5]   8.00-9.00   sec   256 KBytes  2.10 Mbits/sec    0   57.9 KBytes       
[  5]   9.00-10.00  sec   256 KBytes  2.10 Mbits/sec    0   60.6 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.50 MBytes  2.10 Mbits/sec    8             sender
[  5]   0.00-10.18  sec  2.43 MBytes  2.01 Mbits/sec                  receiver

UDP

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 10M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 47837 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   1.00-2.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   2.00-3.00   sec  1.19 MBytes  10.0 Mbits/sec  928  
[  5]   3.00-4.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   4.00-5.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   5.00-6.00   sec  1.19 MBytes  10.0 Mbits/sec  928  
[  5]   6.00-7.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   7.00-8.00   sec  1.19 MBytes  9.99 Mbits/sec  927  
[  5]   8.00-9.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   9.00-10.00  sec  1.19 MBytes  10.0 Mbits/sec  928  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/9273 (0%)  sender
[  5]   0.00-12.75  sec  3.91 MBytes  2.58 Mbits/sec  4.233 ms  6227/9272 (67%)  receiver

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 8M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 41209 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   1.00-2.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   2.00-3.00   sec   975 KBytes  7.99 Mbits/sec  741  
[  5]   3.00-4.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   4.00-5.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   5.00-6.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   6.00-7.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   7.00-8.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   8.00-9.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   9.00-10.00  sec   975 KBytes  7.99 Mbits/sec  741  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  9.54 MBytes  8.00 Mbits/sec  0.000 ms  0/7418 (0%)  sender
[  5]   0.00-13.82  sec  4.70 MBytes  2.86 Mbits/sec  6.924 ms  3759/7418 (51%)  receiver

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 6M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 46235 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   732 KBytes  5.99 Mbits/sec  556  
[  5]   1.00-2.00   sec   733 KBytes  6.01 Mbits/sec  557  
[  5]   2.00-3.00   sec   732 KBytes  6.00 Mbits/sec  556  
[  5]   3.00-4.00   sec   733 KBytes  6.01 Mbits/sec  557  
[  5]   4.00-5.00   sec   732 KBytes  6.00 Mbits/sec  556  
[  5]   5.00-6.00   sec   732 KBytes  6.00 Mbits/sec  556  
[  5]   6.00-7.00   sec   733 KBytes  6.01 Mbits/sec  557  
[  5]   7.00-8.00   sec   732 KBytes  6.00 Mbits/sec  556  
[  5]   8.00-9.00   sec   733 KBytes  6.01 Mbits/sec  557  
[  5]   9.00-10.00  sec   732 KBytes  6.00 Mbits/sec  556  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  7.15 MBytes  6.00 Mbits/sec  0.000 ms  0/5564 (0%)  sender
[  5]   0.00-13.56  sec  3.68 MBytes  2.28 Mbits/sec  8.543 ms  2699/5562 (49%)  receiver

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 4M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 44098 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   1.00-2.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   2.00-3.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   3.00-4.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   4.00-5.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   5.00-6.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   6.00-7.00   sec   488 KBytes  4.01 Mbits/sec  371  
[  5]   7.00-8.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   8.00-9.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   9.00-10.00  sec   487 KBytes  3.99 Mbits/sec  370  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  4.77 MBytes  4.00 Mbits/sec  0.000 ms  0/3709 (0%)  sender
[  5]   0.00-12.04  sec  4.77 MBytes  3.32 Mbits/sec  5.121 ms  2/3709 (0.054%)  receiver

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 2M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 52454 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   245 KBytes  2.01 Mbits/sec  186  
[  5]   1.00-2.00   sec   244 KBytes  1.99 Mbits/sec  185  
[  5]   2.00-3.00   sec   245 KBytes  2.01 Mbits/sec  186  
[  5]   3.00-4.00   sec   244 KBytes  2.00 Mbits/sec  185  
[  5]   4.00-5.00   sec   245 KBytes  2.01 Mbits/sec  186  
[  5]   5.00-6.00   sec   244 KBytes  2.00 Mbits/sec  185  
[  5]   6.00-7.00   sec   245 KBytes  2.00 Mbits/sec  186  
[  5]   7.00-8.00   sec   244 KBytes  2.00 Mbits/sec  185  
[  5]   8.00-9.00   sec   244 KBytes  2.00 Mbits/sec  185  
[  5]   9.00-10.00  sec   245 KBytes  2.01 Mbits/sec  186  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.38 MBytes  2.00 Mbits/sec  0.000 ms  0/1855 (0%)  sender
[  5]   0.00-10.09  sec  2.38 MBytes  1.98 Mbits/sec  6.568 ms  0/1855 (0%)  receiver

Speed ​​without tunnel

$ iperf3 -c 111.111.111.111 -P1 --bitrate 20M -t 20
Connecting to host 111.111.111.111, port 5201
[  5] local 192.168.1.5 port 33892 connected to 111.111.111.111 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.00 MBytes  8.39 Mbits/sec    0   88.2 KBytes       
[  5]   1.00-2.00   sec  1.12 MBytes  9.44 Mbits/sec    1   90.8 KBytes       
[  5]   2.00-3.00   sec  1.12 MBytes  9.43 Mbits/sec    0    105 KBytes       
[  5]   3.00-4.00   sec  1.38 MBytes  11.5 Mbits/sec    0    111 KBytes       
[  5]   4.00-5.00   sec  1.50 MBytes  12.6 Mbits/sec    0    113 KBytes       
[  5]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec    0    116 KBytes       
[  5]   6.00-7.00   sec  1.75 MBytes  14.7 Mbits/sec    0    126 KBytes       
[  5]   7.00-8.00   sec  2.25 MBytes  18.9 Mbits/sec    0    138 KBytes       
[  5]   8.00-9.00   sec  1.75 MBytes  14.7 Mbits/sec    1    116 KBytes       
[  5]   9.00-10.00  sec  2.12 MBytes  17.8 Mbits/sec    0    132 KBytes       
[  5]  10.00-11.00  sec  2.38 MBytes  19.9 Mbits/sec    0    141 KBytes       
[  5]  11.00-12.00  sec  2.38 MBytes  19.9 Mbits/sec    0    142 KBytes       
[  5]  12.00-13.00  sec  2.38 MBytes  19.9 Mbits/sec    1    113 KBytes       
[  5]  13.00-14.00  sec  2.00 MBytes  16.8 Mbits/sec    0    134 KBytes       
[  5]  14.00-15.00  sec  1.88 MBytes  15.7 Mbits/sec    1    107 KBytes       
[  5]  15.00-16.00  sec  1.50 MBytes  12.6 Mbits/sec    0    115 KBytes       
[  5]  16.00-17.00  sec  1.62 MBytes  13.6 Mbits/sec    0    122 KBytes       
[  5]  17.00-18.00  sec  1.62 MBytes  13.6 Mbits/sec    0    132 KBytes       
[  5]  18.00-19.00  sec  2.25 MBytes  18.9 Mbits/sec    0    142 KBytes       
[  5]  19.00-20.00  sec  2.38 MBytes  19.9 Mbits/sec    0    154 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  35.6 MBytes  14.9 Mbits/sec    4             sender
[  5]   0.00-20.08  sec  35.5 MBytes  14.8 Mbits/sec                  receiver

$ iperf3 -c 111.111.111.111 -P1 --bitrate 20M -t 20 -u
Connecting to host 111.111.111.111, port 5201
[  5] local 192.168.1.5 port 46613 connected to 111.111.111.111 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  10.00-11.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  11.00-12.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  12.00-13.00  sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]  13.00-14.00  sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]  14.00-15.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  15.00-16.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  16.00-17.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  17.00-18.00  sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]  18.00-19.00  sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]  19.00-20.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-20.00  sec  47.7 MBytes  20.0 Mbits/sec  0.000 ms  0/37091 (0%)  sender
[  5]   0.00-20.07  sec  46.8 MBytes  19.5 Mbits/sec  0.719 ms  684/37059 (1.8%)  receiver
M0dEx commented 4 months ago

This time I launched the server on another VDS, where the throughput is 100Mbps. The speed dropped. mtu 1400

TCP

$ iperf3 -c 10.0.0.1 -P1 --bitrate 10M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 57852 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   571 KBytes  4.68 Mbits/sec    3   48.7 KBytes       
[  5]   1.00-2.00   sec   317 KBytes  2.60 Mbits/sec    0   52.7 KBytes       
[  5]   2.00-3.00   sec   250 KBytes  2.05 Mbits/sec    2   54.0 KBytes       
[  5]   3.00-4.00   sec   141 KBytes  1.16 Mbits/sec    0   55.3 KBytes       
[  5]   4.00-5.00   sec   214 KBytes  1.75 Mbits/sec    2   56.6 KBytes       
[  5]   5.00-6.00   sec   150 KBytes  1.23 Mbits/sec    3   57.9 KBytes       
[  5]   6.00-7.00   sec   359 KBytes  2.94 Mbits/sec    0   61.9 KBytes       
[  5]   7.00-8.00   sec   298 KBytes  2.44 Mbits/sec    0   64.5 KBytes       
[  5]   8.00-9.00   sec   342 KBytes  2.80 Mbits/sec    0   69.8 KBytes       
[  5]   9.00-10.00  sec   301 KBytes  2.46 Mbits/sec    0   86.9 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.88 MBytes  2.41 Mbits/sec   10             sender
[  5]   0.00-10.29  sec  2.79 MBytes  2.28 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 8M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 46136 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   486 KBytes  3.98 Mbits/sec    3   38.2 KBytes       
[  5]   1.00-2.00   sec   379 KBytes  3.11 Mbits/sec    2   42.1 KBytes       
[  5]   2.00-3.00   sec   254 KBytes  2.08 Mbits/sec    0   46.1 KBytes       
[  5]   3.00-4.00   sec   254 KBytes  2.08 Mbits/sec    0   50.0 KBytes       
[  5]   4.00-5.00   sec   254 KBytes  2.08 Mbits/sec    1   56.6 KBytes       
[  5]   5.00-6.00   sec   251 KBytes  2.06 Mbits/sec    1   57.9 KBytes       
[  5]   6.00-7.00   sec   169 KBytes  1.39 Mbits/sec    1   59.2 KBytes       
[  5]   7.00-8.00   sec   376 KBytes  3.08 Mbits/sec    0   61.9 KBytes       
[  5]   8.00-9.00   sec   234 KBytes  1.92 Mbits/sec    0   67.1 KBytes       
[  5]   9.00-10.00  sec   146 KBytes  1.20 Mbits/sec    0   80.3 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.74 MBytes  2.30 Mbits/sec    8             sender
[  5]   0.00-10.23  sec  2.67 MBytes  2.19 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 6M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 48880 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   512 KBytes  4.19 Mbits/sec    1   80.3 KBytes       
[  5]   1.00-2.00   sec   256 KBytes  2.10 Mbits/sec    4   80.3 KBytes       
[  5]   2.00-3.00   sec   240 KBytes  1.97 Mbits/sec    1   81.6 KBytes       
[  5]   3.00-4.00   sec   265 KBytes  2.17 Mbits/sec    1   82.9 KBytes       
[  5]   4.00-5.00   sec   263 KBytes  2.15 Mbits/sec    0   84.2 KBytes       
[  5]   5.00-6.00   sec   384 KBytes  3.15 Mbits/sec    0   92.1 KBytes       
[  5]   6.00-7.00   sec   256 KBytes  2.10 Mbits/sec    1   69.8 KBytes       
[  5]   7.00-8.00   sec   256 KBytes  2.10 Mbits/sec    0   85.6 KBytes       
[  5]   8.00-9.01   sec   256 KBytes  2.09 Mbits/sec    1   86.9 KBytes       
[  5]   9.01-10.00  sec   256 KBytes  2.11 Mbits/sec    0   89.5 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.88 MBytes  2.41 Mbits/sec    9             sender
[  5]   0.00-10.31  sec  2.69 MBytes  2.19 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 4M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 44136 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   488 KBytes  4.00 Mbits/sec    3   48.7 KBytes       
[  5]   1.00-2.00   sec   272 KBytes  2.23 Mbits/sec    2   51.3 KBytes       
[  5]   2.00-3.00   sec   251 KBytes  2.06 Mbits/sec    1   52.7 KBytes       
[  5]   3.00-4.00   sec   101 KBytes   830 Kbits/sec    0   55.3 KBytes       
[  5]   4.00-5.00   sec   153 KBytes  1.25 Mbits/sec    1   57.9 KBytes       
[  5]   5.00-6.00   sec   251 KBytes  2.06 Mbits/sec    0   60.6 KBytes       
[  5]   6.00-7.00   sec   249 KBytes  2.04 Mbits/sec    0   63.2 KBytes       
[  5]   7.00-8.00   sec   281 KBytes  2.31 Mbits/sec    0   71.1 KBytes       
[  5]   8.00-9.00   sec   256 KBytes  2.10 Mbits/sec    0   88.2 KBytes       
[  5]   9.00-10.00  sec   384 KBytes  3.15 Mbits/sec    0    115 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.62 MBytes  2.20 Mbits/sec    7             sender
[  5]   0.00-10.47  sec  2.49 MBytes  1.99 Mbits/sec                  receiver

$ iperf3 -c 10.0.0.1 -P1 --bitrate 2M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 44124 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   256 KBytes  2.10 Mbits/sec    2   31.6 KBytes       
[  5]   1.00-2.00   sec   256 KBytes  2.10 Mbits/sec    0   35.5 KBytes       
[  5]   2.00-3.00   sec   256 KBytes  2.10 Mbits/sec    0   40.8 KBytes       
[  5]   3.00-4.00   sec   256 KBytes  2.10 Mbits/sec    1   43.4 KBytes       
[  5]   4.00-5.00   sec   256 KBytes  2.10 Mbits/sec    2   46.1 KBytes       
[  5]   5.00-6.00   sec   256 KBytes  2.10 Mbits/sec    2   48.7 KBytes       
[  5]   6.00-7.00   sec   242 KBytes  1.98 Mbits/sec    1   51.3 KBytes       
[  5]   7.00-8.00   sec   270 KBytes  2.21 Mbits/sec    0   55.3 KBytes       
[  5]   8.00-9.00   sec   256 KBytes  2.10 Mbits/sec    0   57.9 KBytes       
[  5]   9.00-10.00  sec   256 KBytes  2.10 Mbits/sec    0   60.6 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.50 MBytes  2.10 Mbits/sec    8             sender
[  5]   0.00-10.18  sec  2.43 MBytes  2.01 Mbits/sec                  receiver

UDP

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 10M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 47837 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   1.00-2.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   2.00-3.00   sec  1.19 MBytes  10.0 Mbits/sec  928  
[  5]   3.00-4.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   4.00-5.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   5.00-6.00   sec  1.19 MBytes  10.0 Mbits/sec  928  
[  5]   6.00-7.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   7.00-8.00   sec  1.19 MBytes  9.99 Mbits/sec  927  
[  5]   8.00-9.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   9.00-10.00  sec  1.19 MBytes  10.0 Mbits/sec  928  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/9273 (0%)  sender
[  5]   0.00-12.75  sec  3.91 MBytes  2.58 Mbits/sec  4.233 ms  6227/9272 (67%)  receiver

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 8M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 41209 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   1.00-2.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   2.00-3.00   sec   975 KBytes  7.99 Mbits/sec  741  
[  5]   3.00-4.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   4.00-5.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   5.00-6.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   6.00-7.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   7.00-8.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   8.00-9.00   sec   977 KBytes  8.00 Mbits/sec  742  
[  5]   9.00-10.00  sec   975 KBytes  7.99 Mbits/sec  741  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  9.54 MBytes  8.00 Mbits/sec  0.000 ms  0/7418 (0%)  sender
[  5]   0.00-13.82  sec  4.70 MBytes  2.86 Mbits/sec  6.924 ms  3759/7418 (51%)  receiver

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 6M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 46235 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   732 KBytes  5.99 Mbits/sec  556  
[  5]   1.00-2.00   sec   733 KBytes  6.01 Mbits/sec  557  
[  5]   2.00-3.00   sec   732 KBytes  6.00 Mbits/sec  556  
[  5]   3.00-4.00   sec   733 KBytes  6.01 Mbits/sec  557  
[  5]   4.00-5.00   sec   732 KBytes  6.00 Mbits/sec  556  
[  5]   5.00-6.00   sec   732 KBytes  6.00 Mbits/sec  556  
[  5]   6.00-7.00   sec   733 KBytes  6.01 Mbits/sec  557  
[  5]   7.00-8.00   sec   732 KBytes  6.00 Mbits/sec  556  
[  5]   8.00-9.00   sec   733 KBytes  6.01 Mbits/sec  557  
[  5]   9.00-10.00  sec   732 KBytes  6.00 Mbits/sec  556  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  7.15 MBytes  6.00 Mbits/sec  0.000 ms  0/5564 (0%)  sender
[  5]   0.00-13.56  sec  3.68 MBytes  2.28 Mbits/sec  8.543 ms  2699/5562 (49%)  receiver

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 4M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 44098 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   1.00-2.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   2.00-3.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   3.00-4.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   4.00-5.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   5.00-6.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   6.00-7.00   sec   488 KBytes  4.01 Mbits/sec  371  
[  5]   7.00-8.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   8.00-9.00   sec   488 KBytes  4.00 Mbits/sec  371  
[  5]   9.00-10.00  sec   487 KBytes  3.99 Mbits/sec  370  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  4.77 MBytes  4.00 Mbits/sec  0.000 ms  0/3709 (0%)  sender
[  5]   0.00-12.04  sec  4.77 MBytes  3.32 Mbits/sec  5.121 ms  2/3709 (0.054%)  receiver

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 2M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 52454 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec   245 KBytes  2.01 Mbits/sec  186  
[  5]   1.00-2.00   sec   244 KBytes  1.99 Mbits/sec  185  
[  5]   2.00-3.00   sec   245 KBytes  2.01 Mbits/sec  186  
[  5]   3.00-4.00   sec   244 KBytes  2.00 Mbits/sec  185  
[  5]   4.00-5.00   sec   245 KBytes  2.01 Mbits/sec  186  
[  5]   5.00-6.00   sec   244 KBytes  2.00 Mbits/sec  185  
[  5]   6.00-7.00   sec   245 KBytes  2.00 Mbits/sec  186  
[  5]   7.00-8.00   sec   244 KBytes  2.00 Mbits/sec  185  
[  5]   8.00-9.00   sec   244 KBytes  2.00 Mbits/sec  185  
[  5]   9.00-10.00  sec   245 KBytes  2.01 Mbits/sec  186  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  2.38 MBytes  2.00 Mbits/sec  0.000 ms  0/1855 (0%)  sender
[  5]   0.00-10.09  sec  2.38 MBytes  1.98 Mbits/sec  6.568 ms  0/1855 (0%)  receiver

Speed ​​without tunnel

$ iperf3 -c 111.111.111.111 -P1 --bitrate 20M -t 20
Connecting to host 111.111.111.111, port 5201
[  5] local 192.168.1.5 port 33892 connected to 111.111.111.111 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  1.00 MBytes  8.39 Mbits/sec    0   88.2 KBytes       
[  5]   1.00-2.00   sec  1.12 MBytes  9.44 Mbits/sec    1   90.8 KBytes       
[  5]   2.00-3.00   sec  1.12 MBytes  9.43 Mbits/sec    0    105 KBytes       
[  5]   3.00-4.00   sec  1.38 MBytes  11.5 Mbits/sec    0    111 KBytes       
[  5]   4.00-5.00   sec  1.50 MBytes  12.6 Mbits/sec    0    113 KBytes       
[  5]   5.00-6.00   sec  1.25 MBytes  10.5 Mbits/sec    0    116 KBytes       
[  5]   6.00-7.00   sec  1.75 MBytes  14.7 Mbits/sec    0    126 KBytes       
[  5]   7.00-8.00   sec  2.25 MBytes  18.9 Mbits/sec    0    138 KBytes       
[  5]   8.00-9.00   sec  1.75 MBytes  14.7 Mbits/sec    1    116 KBytes       
[  5]   9.00-10.00  sec  2.12 MBytes  17.8 Mbits/sec    0    132 KBytes       
[  5]  10.00-11.00  sec  2.38 MBytes  19.9 Mbits/sec    0    141 KBytes       
[  5]  11.00-12.00  sec  2.38 MBytes  19.9 Mbits/sec    0    142 KBytes       
[  5]  12.00-13.00  sec  2.38 MBytes  19.9 Mbits/sec    1    113 KBytes       
[  5]  13.00-14.00  sec  2.00 MBytes  16.8 Mbits/sec    0    134 KBytes       
[  5]  14.00-15.00  sec  1.88 MBytes  15.7 Mbits/sec    1    107 KBytes       
[  5]  15.00-16.00  sec  1.50 MBytes  12.6 Mbits/sec    0    115 KBytes       
[  5]  16.00-17.00  sec  1.62 MBytes  13.6 Mbits/sec    0    122 KBytes       
[  5]  17.00-18.00  sec  1.62 MBytes  13.6 Mbits/sec    0    132 KBytes       
[  5]  18.00-19.00  sec  2.25 MBytes  18.9 Mbits/sec    0    142 KBytes       
[  5]  19.00-20.00  sec  2.38 MBytes  19.9 Mbits/sec    0    154 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  35.6 MBytes  14.9 Mbits/sec    4             sender
[  5]   0.00-20.08  sec  35.5 MBytes  14.8 Mbits/sec                  receiver

$ iperf3 -c 111.111.111.111 -P1 --bitrate 20M -t 20 -u
Connecting to host 111.111.111.111, port 5201
[  5] local 192.168.1.5 port 46613 connected to 111.111.111.111 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   1.00-2.00   sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]   2.00-3.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   3.00-4.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   4.00-5.00   sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]   5.00-6.00   sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]   6.00-7.00   sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]   7.00-8.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   8.00-9.00   sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]   9.00-10.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  10.00-11.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  11.00-12.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  12.00-13.00  sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]  13.00-14.00  sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]  14.00-15.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  15.00-16.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  16.00-17.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
[  5]  17.00-18.00  sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]  18.00-19.00  sec  2.38 MBytes  20.0 Mbits/sec  1854  
[  5]  19.00-20.00  sec  2.38 MBytes  20.0 Mbits/sec  1855  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-20.00  sec  47.7 MBytes  20.0 Mbits/sec  0.000 ms  0/37091 (0%)  sender
[  5]   0.00-20.07  sec  46.8 MBytes  19.5 Mbits/sec  0.719 ms  684/37059 (1.8%)  receiver

What is the latency between the two hosts?

vukitoso commented 4 months ago

What is the latency between the two hosts?

Without tunnel: 20 packets transmitted, 20 received, 0% packet loss, time 19019ms rtt min/avg/max/mdev = 29.285/44.879/68.074/9.862 ms

quincy tunnel: 20 packets transmitted, 20 received, 0% packet loss, time 19022ms rtt min/avg/max/mdev = 38.022/50.584/78.523/11.309 ms

M0dEx commented 4 months ago

@vukitoso I'm sorry, but at this time, I can't find any reason for this behaviour other than the VDS provider throttling UDP traffic, and I cannot replicate it locally with virtual machines and simulating similar network conditions.

I'll keep this open in case you have any updates or if I succeed in replicating this issue.

vukitoso commented 4 months ago

Yes, I also noticed that the problem with a decrease in bitrate, an increase in ping and packet loss occurs only on my LTE Internet provider. In other cases the problem is not reproduced.

M0dEx commented 3 months ago

@vukitoso Is everything working well so far? If so, do you think this issue can be closed?

vukitoso commented 3 months ago

On an LTE connection, I decided to abandon UDP tunnels.

On version 0.8.3: Right now I checked it on two VDS located in data centers. There are no losses at 10 Mbit/s.

$ iperf3 -c 10.0.0.1 -P1 -u --bitrate 10M
Connecting to host 10.0.0.1, port 5201
[  5] local 10.0.0.2 port 58942 connected to 10.0.0.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   1.00-2.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   2.00-3.00   sec  1.19 MBytes  10.0 Mbits/sec  928  
[  5]   3.00-4.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   4.00-5.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   5.00-6.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   6.00-7.00   sec  1.19 MBytes  10.0 Mbits/sec  928  
[  5]   7.00-8.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   8.00-9.00   sec  1.19 MBytes  10.0 Mbits/sec  927  
[  5]   9.00-10.00  sec  1.19 MBytes  10.0 Mbits/sec  928  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-10.00  sec  11.9 MBytes  10.0 Mbits/sec  0.000 ms  0/9273 (0%)  sender
[  5]   0.00-10.43  sec  11.9 MBytes  9.58 Mbits/sec  1.584 ms  0/9273 (0%)  receiver
M0dEx commented 3 months ago

Great to hear that. That means this issue can be closed, right?

vukitoso commented 3 months ago

yes, thx.