search

Ysurac / openmptcprouter

OpenMPTCProuter is an open source solution to aggregate multiple internet connections using Multipath TCP (MPTCP) on OpenWrt
https://www.openmptcprouter.com/
GNU General Public License v3.0
1.92k stars 274 forks source link

100 + 100 = 100 Mb/s #3026

Closed adamb closed 9 months ago

adamb commented 1 year ago

Expected Behavior

I have two network connections that are about 100 Mb/s each when I test them directly. LTE modem and Starlink.

When I put them together on OpenMPTCProuter I get about the same. I was expecting more like 200 Mbs.

Current Behavior

My MPTCP connections seems slower or the same and the individual WAN connections.

Specifications

I have a Raspbery Pi4. I'm using UE300 USB3 to ethernet connections.

I have tried various VPS providers. Linode, OneProvider, and even my own server at my second home where I have gigabit fiber.

omr-test-speed directly on the Pi and Ookla on my laptop give me about the same results when comparing the full system.

But when I use omr-test-speed eth1 it's much slower than if I connect directly to the LTE modem. About 1/4th the speed when measured with omr-test-speeed eth1

The starlink connections does better with the omr-test-speed eht2. It seems to give me over 100 Mb/s.

Ysurac commented 1 year ago

It's not a good idea to use USB device on RPI, they are always bad with USB devices. Do you use a powered USB switch ?

adamb commented 1 year ago

Since the Pi has just one Ethernet port, should I connect everything to that using an Ethernet switch?

Right now I have the built in connected to my home wifi. One USB3 converts to an Ethernet for the LTE and another USB3 to Ethernet for the Starlink.

Ysurac commented 1 year ago

Yes, it's better to connect all to an Ethernet switch. RPI then use macvlan to create "virtual" interfaces on the real interface. Else USB may work with a powered USB switch.

adamb commented 1 year ago

Ah, I have Starlink and it has to be DHCP. And I need a physical eth2 in that case, correct? So a VLAN wouldn't work.

Also, I have to figure out what a vlan is and how to configure it. :-)

Ysurac commented 1 year ago

If you use only fixed IP, you don't care Or you need another router/box to put between Starlink to disable DHCP. Or else you can use VLAN but you need a manageable ethernet switch. Or else use only one USB ethernet, maybe this will work, else use a powered USB switch.

I would try to only have Starlink on USB for now. If the speed with only one USB used is ok, go with it.

adamb commented 1 year ago

My switch doesn't support vlans. But if I plug the LTE and the Pi into that with static IPs in the wizard. (Both using eth0) then I get an error. Network interface duplicated.

I have a fixed IP on the LTE modem. 

# ip a | grep eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 100
    inet 192.168.100.1/24 brd 192.168.100.255 scope global eth0
    inet 192.168.1.2/24 brd 192.168.1.255 scope global eth0

Is that what you meant by "If you use only fixed IP, you don't care"?

adamb commented 1 year ago

Ah, I get it. I missed the MacVLAN thing. Thanks!

adamb commented 1 year ago

With the MacVLAN for my 5G connection and eth2 (USB3 with the UE300) being my Starlink connection, I'm not getting great performance. I'm seeing less performance through OMR using two connections than using just Starlink and no OMR. Any suggestions on how to debug the performance? My 5G connection gets around 50-100 Mb/s and Starlink usually 100-200 Mb/s by themselves.

How can I find the bottleneck? Is it the Pi? Is it the VPS?

Would it help to use IPV6?

Ysurac commented 1 year ago

Check with omr-test-speed <interface> if all interface give the correct speed. If all is ok, use omr-test-speed on the VPS to check VPS speed.

adamb commented 1 year ago

I ran some more tests. the combined speed (omr-test-speed) is slower than the slowest single connection. The VPS is plenty fast. I'm quite surprised. I really thought the combined speed would be closer to eth1+eth2 speeds.

eth1 is TMobile eth2 is Starlink

omr-test-speed # on the rpi4
omr-test-speed eth1
omr-test-speed eth2
omr-test-speed # on the vps

             | curl | Mb/s | curl | Mb/s
   combined | 4348k | 34.0 | 3852k | 30.1
       eth1 | 17.7M | 141.6 | 3865k | 30.2
       eth2 | 20.8M | 166.4 | 17.3M | 138.4
        vps | 534M  | 4272.0 | 9536M | 76288.0

I'm really surprised. Is this expected?

Ysurac commented 1 year ago

Yes there is a problem here. All is green in status page ? What is the latency indicated here ? For information, the RPI4 can provide at least 550Mb/s with encryption enabled (by default), so it's not the problem.

adamb commented 1 year ago

Latency looks good to me.

image

Here's top. Not a lot going on.

image

Connections doesn't seem crazy.

image

Plenty of disk space:

root@OpenMPTCProuter:~# df -h
Filesystem                Size      Used Available Use% Mounted on
/dev/root               503.6M    246.7M    246.6M  50% /rom
devtmpfs                896.9M         0    896.9M   0% /rom/dev
tmpfs                   929.2M      1.5M    927.7M   0% /tmp
/dev/mmcblk0p2          503.6M    246.7M    246.6M  50% /
/dev/mmcblk0p1           63.9M     16.4M     47.4M  26% /boot
tmpfs                   512.0K         0    512.0K   0% /dev

Anything else I should check?

Ysurac commented 1 year ago

Latency is quite high but ok. Do you use SQM in wizard ? If yes, try to disable it. Check that BBRv2 is used in Network->MPTCP and if it's the case, try Cubic. You can also try another VPS

adamb commented 1 year ago

SQM off and BBRv2 seemed to do the trick! Much better now. I had SQM on and Cubic before.

image
Ysurac commented 1 year ago

BBRv2 help on speed, SQM help on connection saturation. If SQM is not correctly configured you can get bad results. What was your SQM configuration ? Only enabled without download/upload speed ?

adamb commented 1 year ago

I'm not sure what it was set to. I just used the defaults. It was on, and I turned it off wherever I saw it on.

Thanks for your help!

github-actions[bot] commented 9 months ago

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days

jhenkens commented 4 months ago

Just sharing some more datapoints:

DSL + LTE/5g (comes and goes). I had it working great back in January, and use it with a VPN from VPS -> Residential FTTH, and at some point it stopped actually aggregating the speeds, and just behaved as a failover. I recreated VPS using 6.1 kernel, and recreated the router as 6.1 kernel as well, and reapplied settings. It still wasn't aggregating. I poked around in a lot of settings, but this post finally helped.

I'm currently getting good aggregated speeds, and these were the settings I changed. Not sure if all or some are needed. I believe I had tried SQM both on and off before changing the other things, to no effect.

  1. SQM on both WANs Enabled -> Disabled (same with autorotate)
  2. Congestion Control BBR3 -> Cubic
  3. Master Interface Change Balancing -> Dynamic (not sure what this one does, really, but was suggested elsewhere).