search

angt / glorytun

Multipath UDP tunnel
BSD 2-Clause "Simplified" License
672 stars 102 forks source link

Throughput to less #31

Closed d3dx9 closed 5 years ago

d3dx9 commented 5 years ago

Hi,

so actually I'm trying to find a solution to bundle 2 connections in the range of 800Mbit in the end. Actually I get a really bad throughput with AES and ChaCha20.

Client:

root@bondingrouter:~# glorytun bench

  bench      aes256gcm
  libsodium  1.0.11

  precision  2^(-10)
  bufsize    65536 bytes
  duration   1 second

------------------------------------------------------------
 2^n       min            avg            max          delta
------------------------------------------------------------
   0     49.29 Mbps     49.42 Mbps     49.51 Mbps  8.75e-03
   1     97.62 Mbps     97.62 Mbps     98.01 Mbps  3.29e-02
   2    194.38 Mbps    194.39 Mbps    195.03 Mbps  1.39e-02
   3    377.57 Mbps    381.62 Mbps    381.87 Mbps  2.41e-01
   4    770.51 Mbps    770.53 Mbps    770.53 Mbps  1.60e-02
   5   1432.62 Mbps   1440.98 Mbps   1440.98 Mbps  1.04e-01
   6   2358.86 Mbps   2372.13 Mbps   2385.08 Mbps  1.35e+00
   7   6132.72 Mbps   6132.72 Mbps   6165.24 Mbps  5.92e+00
   8   7979.00 Mbps   8112.97 Mbps   8112.97 Mbps  7.81e+00
   9   9754.27 Mbps   9795.36 Mbps   9799.62 Mbps  4.26e+00
  10  10826.03 Mbps  10826.03 Mbps  10829.40 Mbps  3.37e+00
  11  11449.07 Mbps  11490.85 Mbps  11550.91 Mbps  3.30e+00
  12  11801.81 Mbps  11801.81 Mbps  11805.90 Mbps  4.09e+00
  13  12041.63 Mbps  12042.88 Mbps  12083.37 Mbps  1.25e+00
  14  12132.29 Mbps  12177.17 Mbps  12190.60 Mbps  7.07e+00
  15  12233.86 Mbps  12234.90 Mbps  12234.90 Mbps  1.04e+00
  16  11988.61 Mbps  12202.87 Mbps  12208.37 Mbps  5.50e+00

Server:

root@AMS-123450:~/MLVPN/src# glorytun bench

  bench      aes256gcm
  libsodium  1.0.11

  precision  2^(-10)
  bufsize    65536 bytes
  duration   1 second

------------------------------------------------------------
 2^n       min            avg            max          delta
------------------------------------------------------------
   0      8.68 Mbps      8.68 Mbps      8.68 Mbps  1.37e-03
   1     17.17 Mbps     17.17 Mbps     17.17 Mbps  4.13e-04
   2     34.09 Mbps     34.09 Mbps     34.09 Mbps  3.37e-04
   3     67.20 Mbps     67.20 Mbps     67.21 Mbps  1.17e-02
   4    133.10 Mbps    133.10 Mbps    133.16 Mbps  6.08e-02
   5    246.12 Mbps    246.14 Mbps    246.14 Mbps  1.10e-02
   6    433.42 Mbps    433.42 Mbps    433.43 Mbps  6.75e-03
   7    802.88 Mbps    802.93 Mbps    802.93 Mbps  5.43e-02
   8    947.08 Mbps    947.08 Mbps    947.22 Mbps  1.44e-01
   9   1055.08 Mbps   1055.10 Mbps   1055.10 Mbps  2.10e-02
  10   1116.01 Mbps   1116.01 Mbps   1116.03 Mbps  1.28e-02
  11   1149.03 Mbps   1149.04 Mbps   1149.04 Mbps  6.32e-03
  12   1166.41 Mbps   1166.43 Mbps   1166.43 Mbps  2.40e-02
  13   1175.18 Mbps   1175.18 Mbps   1175.34 Mbps  1.61e-01
  14   1179.62 Mbps   1179.62 Mbps   1179.67 Mbps  4.72e-02
  15   1181.58 Mbps   1181.62 Mbps   1181.62 Mbps  3.60e-02
  16   1182.71 Mbps   1182.78 Mbps   1182.78 Mbps  7.29e-02

The client is a virtual machine. The server side is a dedicated server with 2 cores but AES enabled. (Intel Avoton C2350).

While benching with iperf3 I get really bad results 

root@bondingrouter:~# iperf3 -c 10.0.1.1 -R
Connecting to host 10.0.1.1, port 5201
Reverse mode, remote host 10.0.1.1 is sending
[  4] local 10.0.1.2 port 53852 connected to 10.0.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  3.94 MBytes  33.1 Mbits/sec
[  4]   1.00-2.00   sec  15.9 MBytes   133 Mbits/sec
[  4]   2.00-3.00   sec  24.4 MBytes   205 Mbits/sec
[  4]   3.00-4.00   sec  25.4 MBytes   213 Mbits/sec
[  4]   4.00-5.00   sec  26.2 MBytes   220 Mbits/sec
[  4]   5.00-6.00   sec  25.2 MBytes   212 Mbits/sec
[  4]   6.00-7.00   sec  25.4 MBytes   213 Mbits/sec
[  4]   6.00-7.00   sec  25.4 MBytes   213 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -

Actually it's just tested with one line which has 500Mbit, but I would say, that still wouldn't hit the 800Mbit with both lines. Any way to improve that to get around 800Mbit through the tunnel?

Kind regards

angt commented 5 years ago

Hi, Your server looks very slow, iperf -s and glorytun maybe kills your perf. Also, the code is absolutely not optimized. I wait for a stable and well tested release before working on perf. On my setup with good hardware I touch 1Gbit/s.

angt commented 5 years ago

If you put theiperf3 on a second server do you have the same results ?

d3dx9 commented 5 years ago

Hi,

I only have a KVM without AES support. With ChaCha20 and AES I also only reach on that server around 200Mbit. Without tunnel on both servers I can reach the full 500Mbit.

LXC without AES as server (16 cores)

Connecting to host 10.0.1.1, port 5201
Reverse mode, remote host 10.0.1.1 is sending
[  4] local 10.0.1.2 port 53910 connected to 10.0.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  26.3 MBytes   221 Mbits/sec
[  4]   1.00-2.00   sec  18.4 MBytes   154 Mbits/sec
[  4]   2.00-3.00   sec  19.7 MBytes   165 Mbits/sec
[  4]   3.00-4.00   sec  20.5 MBytes   172 Mbits/sec
[  4]   4.00-5.00   sec  21.3 MBytes   179 Mbits/sec
[  4]   5.00-6.00   sec  18.5 MBytes   155 Mbits/sec
[  4]   6.00-7.00   sec  16.3 MBytes   137 Mbits/sec
[  4]   7.00-8.00   sec  17.5 MBytes   147 Mbits/sec
[  4]   8.00-9.00   sec  17.6 MBytes   148 Mbits/sec
[  4]   8.00-9.00   sec  17.6 MBytes   148 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -

That would be the speeds without tunneling

d3dx9@bondingrouter:~$ iperf3 -c 185.188.4.5 -R
Connecting to host 185.188.4.5, port 5201
Reverse mode, remote host 185.188.4.5 is sending
[  4] local 10.0.5.207 port 58738 connected to 185.188.4.5 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  49.1 MBytes   412 Mbits/sec
[  4]   1.00-2.00   sec  60.0 MBytes   503 Mbits/sec
[  4]   2.00-3.00   sec  58.9 MBytes   494 Mbits/sec
[  4]   3.00-4.00   sec  59.9 MBytes   503 Mbits/sec
[  4]   4.00-5.00   sec  59.4 MBytes   498 Mbits/sec
[  4]   5.00-6.00   sec  59.6 MBytes   500 Mbits/sec
[  4]   6.00-7.00   sec  56.6 MBytes   475 Mbits/sec
[  4]   7.00-8.00   sec  57.8 MBytes   485 Mbits/sec
[  4]   8.00-9.00   sec  57.7 MBytes   484 Mbits/sec
[  4]   9.00-10.00  sec  58.2 MBytes   488 Mbits/sec

So I can't really tell where the problems come from.

d3dx9 commented 5 years ago

So found another unexpected behavior, the speeds dropping with http downloads from ~20mb/s after like 10 seconds to 3mb/s.

angt commented 5 years ago

Can you show me the output of glorytun path ?

d3dx9 commented 5 years ago 
path UP
  status:   DEGRADED
  bind:     10.1.1.1 port 5000
  public:   - port 23875
  peer:     185.188.4.5 port 5000
  mtu:      1302 bytes
  rtt:      0.000 ms
  rttvar:   0.000 ms
  rate tx:  6250000 bytes/sec
  rate rx:  62500000 bytes/sec
  total tx: 5 packets
  total rx: 0 packets
angt commented 5 years ago

Your tunnel is dead now :(

d3dx9 commented 5 years ago

Sorry, that was my fault. The glorytun was killed on serverside while I was still on work.

root@bondingrouter:/home/d3dx9# glorytun path
path UP
  status:   OK
  bind:     10.1.1.1 port 5000
  public:   homeip port 8776
  peer:     serverip port 5000
  mtu:      1387 bytes
  rtt:      32.547 ms
  rttvar:   13.241 ms
  rate tx:  6250000 bytes/sec
  rate rx:  62500000 bytes/sec
  total tx: 30639 packets
  total rx: 2 packets
angt commented 5 years ago

Ok can you check the output of ip route get SERVER_IP and check if it goes to glorytun ? If you want we can test with one of my server, contact me at adrien at gallouet dot fr.

d3dx9 commented 5 years ago

Yes, it goes via glorytun. Ill drop you a message.

angt commented 5 years ago

After some test I confirm I cannot reach 1Gbit/s either. I'll dig in the next days to see what's happened.

Sorry for the inconvenience.

angt commented 5 years ago

Could you do the same test in UDP like iperf3 -c ... -u -b 480M -R ?

d3dx9 commented 5 years ago

Serverside:

Accepted connection from 10.0.1.2, port 54068
[  5] local 10.0.1.1 port 5201 connected to 10.0.1.2 port 52064
[ ID] Interval           Transfer     Bandwidth       Total Datagrams
[  5]   0.00-1.00   sec  52.3 MBytes   439 Mbits/sec  6697
[  5]   1.00-2.00   sec  57.3 MBytes   480 Mbits/sec  7329
[  5]   2.00-3.00   sec  57.2 MBytes   480 Mbits/sec  7320
[  5]   3.00-4.00   sec  57.0 MBytes   478 Mbits/sec  7294
[  5]   4.00-5.00   sec  57.4 MBytes   481 Mbits/sec  7341
[  5]   5.00-6.00   sec  57.2 MBytes   480 Mbits/sec  7318
[  5]   6.00-7.00   sec  57.1 MBytes   479 Mbits/sec  7308
[  5]   7.00-8.00   sec  57.5 MBytes   482 Mbits/sec  7359
[  5]   7.00-8.00   sec  57.5 MBytes   482 Mbits/sec  7359
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  5]   0.00-8.00   sec   487 MBytes   511 Mbits/sec  0.000 ms  0/62326 (0%)

Clientside:

root@bondingrouter:~# iperf3 -c 10.0.1.1 -u -b 480M -R
Connecting to host 10.0.1.1, port 5201
Reverse mode, remote host 10.0.1.1 is sending
[  4] local 10.0.1.2 port 52064 connected to 10.0.1.1 port 5201
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-1.00   sec  10.2 MBytes  85.4 Mbits/sec  0.158 ms  6084/7387 (82%) 
[  4]   1.00-2.00   sec  9.84 MBytes  82.5 Mbits/sec  0.210 ms  6103/7362 (83%) 
[  4]   2.00-3.00   sec  9.69 MBytes  81.3 Mbits/sec  0.718 ms  6088/7328 (83%) 
[  4]   3.00-4.00   sec  10.3 MBytes  86.1 Mbits/sec  0.178 ms  5926/7240 (82%) 
[  4]   4.00-5.00   sec  10.9 MBytes  91.2 Mbits/sec  0.087 ms  5979/7371 (81%) 
[  4]   5.00-6.00   sec  11.5 MBytes  96.7 Mbits/sec  0.116 ms  5851/7326 (80%) 
[  4]   6.00-7.00   sec  10.3 MBytes  86.7 Mbits/sec  0.153 ms  6027/7350 (82%) 
[  4]   7.00-8.00   sec  10.2 MBytes  85.7 Mbits/sec  0.164 ms  6016/7324 (82%) 
^C[  4]   8.00-8.51   sec  5.09 MBytes  83.9 Mbits/sec  0.106 ms  2970/3622 (82%)
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Jitter    Lost/Total Datagrams
[  4]   0.00-8.51   sec  0.00 Bytes  0.00 bits/sec  0.106 ms  51044/62310 (82%) 
[  4] Sent 62310 datagrams
iperf3: interrupt - the client has terminated

For me it seems it's affected by the MTU because of dropping speeds while using iperf3 without extra args.

d3dx9 commented 5 years ago

You might have an update for that issue?

0xgrm commented 5 years ago

Would be interested too, I have big speed problems with glorytun-udp that I think are related to MTU. I have to use glorytun-tcp as of now (on OpenMPTCProuter).

angt commented 5 years ago

With the last release I'm able to get more than 1Gbps with TCP when the hardware is capable enough and by tweeking the txqueuelen/limit of the interface/qdisc.

Connecting to host 10.166.178.2, port 5201
Reverse mode, remote host 10.166.178.2 is sending
[  5] local 10.166.178.3 port 54730 connected to 10.166.178.2 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   172 MBytes  1.44 Gbits/sec
[  5]   1.00-2.00   sec   174 MBytes  1.46 Gbits/sec
[  5]   2.00-3.00   sec   174 MBytes  1.46 Gbits/sec
[  5]   3.00-4.00   sec   175 MBytes  1.47 Gbits/sec
[  5]   4.00-5.00   sec   180 MBytes  1.51 Gbits/sec
[  5]   5.00-6.00   sec   181 MBytes  1.51 Gbits/sec
[  5]   6.00-7.00   sec   146 MBytes  1.23 Gbits/sec                  
[  5]   7.00-8.00   sec   178 MBytes  1.49 Gbits/sec                  
[  5]   8.00-9.00   sec   178 MBytes  1.49 Gbits/sec                  
[  5]   9.00-10.00  sec   177 MBytes  1.48 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.04  sec  1.70 GBytes  1.45 Gbits/sec  2514             sender
[  5]   0.00-10.00  sec  1.69 GBytes  1.45 Gbits/sec                  receiver

iperf Done.

$ iperf3 -c 10.166.178.2
Connecting to host 10.166.178.2, port 5201
[  5] local 10.166.178.3 port 55120 connected to 10.166.178.2 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   153 MBytes  1.29 Gbits/sec  264    433 KBytes
[  5]   1.00-2.00   sec   158 MBytes  1.33 Gbits/sec  122    362 KBytes
[  5]   2.00-3.00   sec   138 MBytes  1.16 Gbits/sec   88    451 KBytes
[  5]   3.00-4.00   sec   161 MBytes  1.35 Gbits/sec  240    332 KBytes
[  5]   4.00-5.00   sec   158 MBytes  1.33 Gbits/sec   96    448 KBytes
[  5]   5.00-6.00   sec   163 MBytes  1.37 Gbits/sec  250    436 KBytes
[  5]   6.00-7.00   sec   155 MBytes  1.30 Gbits/sec  168    411 KBytes
[  5]   7.00-8.00   sec   148 MBytes  1.24 Gbits/sec  153    303 KBytes
[  5]   8.00-9.00   sec   162 MBytes  1.36 Gbits/sec  146    399 KBytes
[  5]   9.00-10.00  sec   165 MBytes  1.39 Gbits/sec    0    629 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.52 GBytes  1.31 Gbits/sec  1527             sender
[  5]   0.00-10.04  sec  1.52 GBytes  1.30 Gbits/sec                  receiver

iperf Done.

with this setup

qdisc fq_codel 803a: root refcnt 2 limit 20000p flows 1024 quantum 1500 target 5.0ms interval 100.0ms memory_limit 32Mb ecn

I'm closing the issue, do not hesitate to open a new one if the problem persists :)