search

Ysurac / openmptcprouter

OpenMPTCProuter is an open source solution to aggregate multiple internet connections using Multipath TCP (MPTCP) on OpenWrt
https://www.openmptcprouter.com/
GNU General Public License v3.0
1.8k stars 258 forks source link

Massive jitter #3411

Open Funstar81 opened 3 months ago

Funstar81 commented 3 months ago

Expected Behavior

Having a stable connection without great jitter.

Current Behavior

Having much jitter, Download sometimes not good, Upload always as expected.

Steps to Reproduce the Problem

  1. Ping 10.255.252.1

Context (Environment)

WAN1 - Vodafone Cable 50 MBit/s down, 25 MBit/s up, 22ms, 1500 MTU WAN2 - TNG Fibre 1000 MBit/s down, 100 MBit/s up, 19ms, 1492 MTU (PPPoE)

--- 10.255.252.1 ping statistics --- 1920 packets transmitted, 1920 received, 0% packet loss, time 1921696ms rtt min/avg/max/mdev = 20.541/155.405/2925.251/325.697 ms, pipe 3

Specifications

Funstar81 commented 3 months ago

I disconnected Vodafone to try if it gets any better, but made no change except that the whole tunnel went town/my Citrix session hung. It came back online a few seconds later, but I was hoping for no impact as long as at least one uplink is available.

Funstar81 commented 3 months ago

Another strange thing: I started iperf3 server on the vps and did a speedtest from my local machine to the vps. I only get 5 MBit/s downstream. When I do speedtest.net, I get several hundred MBit/s.

UPDATE: Two minutes later, speedtest.net shows less than 5 MBit/s aswell.

Funstar81 commented 3 months ago

This gets more and more interesting. I started a download on my companys laptop. With only 400 KByte/sec, it was awfully slow. Then I made a speedtest.net on my private PC, which showed 50 MBit/s (which is not good either) but after that, the download on my companys laptop went up to 1.300 KByte/sec where it was 400 KByte/sec before.

Ysurac commented 3 months ago

What latency do you have between your connections and the VPS ? (it's displayed in status page) It's an update or a new install ? What is the proxy/VPN you are using ?

Funstar81 commented 3 months ago

Latency is as noticed in context section. Vodafone (Cable) is 24ms, with TNG (fibre) it is 19ms.

I installed everything from scratch several times today. VPS, OMR router, even my OPNsense firewall (which is the only client to the OMR router), everything is fresh now.

Proxy is the default one. (Shadowsocks-Rust 2022 I guess)

Ysurac commented 3 months ago

On OpenMPTCProuter, ping command only test ICMP and this use the VPN like UDP (TCP use the proxy that is faster). Ping is always high or only sometimes ? You are not doing others traffic at same time ?

Funstar81 commented 3 months ago

Those were my stats from today: --- 10.255.252.1 ping statistics --- 1920 packets transmitted, 1920 received, 0% packet loss, time 1921696ms rtt min/avg/max/mdev = 20.541/155.405/2925.251/325.697 ms, pipe 3

As you can see, the min is nice with 20ms, but the average of more than 1900 pings was 155ms, which is way too high. Right now, ping to Google DNS is fine: --- 8.8.8.8 ping statistics --- 115 packets transmitted, 115 received, 0% packet loss, time 114202ms rtt min/avg/max/mdev = 21.250/24.469/53.072/4.321 ms

Will booting up my companys notebook again, lets see if the VPN client is causing struggle, cause it was online all day.

Funstar81 commented 3 months ago

So with the VPN activated on my companys notebook, I can see a higher ping here and there around 100ms, but nothing to worry about. During the day I saw pings up to 17 seconds (not ms, s!). Video calls, phone calls and Citrix sessions and even the whole VPN tunnel (Cisco) were flapping. Will monitor tommorrow and report again.

Funstar81 commented 3 months ago

UPDATE: My wife just started her companys laptop, connected via VPN (Cisco aswell) and the latency of my ping goes straight up. While I had around 25ms with my laptop shut down, the latency is now up to 400ms:

--- 8.8.8.8 ping statistics --- 100 packets transmitted, 100 received, 0% packet loss, time 99144ms rtt min/avg/max/mdev = 21.965/257.105/649.008/172.686 ms

She is opening pictures from a remote fileshare (SMB).

Ysurac commented 3 months ago

You can test others VPNs in System->OpenMPTCProuter, Wizard tab, "Advanced settings" checkbox. Maybe another one can be better in your case.

Brazzo978 commented 3 months ago

Hi @Funstar81 i have the same issue just that the vpn goes down interely , basically to replicate that just spin un an omr install with decent bandwidth (200mb its fine ) then enable openvpn tcp / glorytun tcp. And on a client run a wireguard client, do a simple speedtest and the vpn will drop always , tested on 3 different install with different connection and when one of the connection gets ping of around 100ms to the vps the tunnel weirdly has 10/20s of ping , one thing that you can do is setting on omr tracker the maximum delay for the tunnel and also in glorytun setting put something crazy like 30s as maximum delay , the tunnel has an extreme delay but traffic still gets trough , not at an optimal speed but still , another thing I found out can help is in sqm remove autorate from the tunnel setting and try changing sqm algorithm for the tunnel shaping too , if you use open/pfsense under omr in that state don't ever use ping to an external ip as gateway test cause it's gona flag omr down even if it's not cause omr passes ping in the vpn that now has 20s of rtt ahahahah , also the snapshot for me it's working really better , also in the snapshot v2ray / xray port forwarding is back working so you can ( one port for each rule ) port forward over proxy and also udp over proxy

Funstar81 commented 3 months ago

Thank you for your feedback. Got it running right now with Glorytun UDP and without SQM. Connection stable right now, no AnnyConnect disconnect so far and great latency to 10.255.252.1 even while having a video call AND doing a speedtest with best values so far.

Funstar81 commented 3 months ago

--- 10.255.252.1 ping statistics --- 11401 packets transmitted, 11320 received, 0.710464% packet loss, time 11421551ms rtt min/avg/max/mdev = 14.948/23.548/2972.935/55.930 ms, pipe 3

Brazzo978 commented 3 months ago

Can you share the setting you found out best for you ?

Funstar81 commented 3 months ago

Everything on default (Shadowsocks-Rust 2022, AES-256-CGM) but as VPN I switched from OpenVPN to GloryTun UDP and deactivated SQM on my uplinks. Speed is calculated by speedtest.

Just did a speedtest again with 921 down and 125 up, so full speed. Ping latency stays cool at around 30ms, no disconnect on my AnyConnect VPN.

Funstar81 commented 3 months ago

As "OpenVPN TCP" was selected as default, TCP Meltdown probably was the cause of this problem. See https://blog.vpntracker.com/why-vpn-tunneling-over-tcp-sucks/ for more details. So if UDP is better than TCP for VPN tunnels (as described in the link provided), a UDP based VPN should be the default setting in OMR.

Ysurac commented 3 months ago

Glorytun UDP doesn't always work correctly and in many case doesn't offer a good result, it's why it's not the default.

Funstar81 commented 3 months ago

So what about OpenVPN UDP instead of OpenVPN TCP?

Ysurac commented 3 months ago

OpenVPN UDP can't be used for aggregation.

Funstar81 commented 3 months ago

So we finally agree that a UDP-based VPN generally would be better for both performance (due to lower overhead) and stability (avoid TCP Meltdown), but not necessarily GloryTun UDP is the best solution, because in your experience this can also lead to problems.

In this case, we can close this issue. Hopefully it helps others that need to have a company VPN connection like AnyConnect inside of their OMR tunnel.

Brazzo978 commented 2 months ago

but if i understand correctly glorytun udp can aggregate udp using its internal method but there is a problem with a lot of packet that are received out of order , glorytun creator never fixed that issue as i am aware of , am i right @Ysurac @Funstar81