How to setup redundant connection? not aggregation.

[openaspace]

HI need to use the system for live streaming events using LTE usb key.

From my tests the connection switch from one to another data sim or cable connection it's not without interruption. Some packets are lost and some times it employ from 5 to 10 seconds to make the switch... this mean that it's not working in redundant mode, but in aggregation mode.

I need to have the clone of the same signal on all connections in the the tunnel , to not loose in any case the connection.

I'm not understanding how to set this in explicit way. In the control panel the only parameter I see is in the "Master interface selection" with "balance" "Dynamic changes" etc..

But how to set a real realtime backup without loosing 1ms of pachets?

Really thank you.

[Ysurac]

It's not the same issue.

[openaspace]

This should work with USB dongle, there is no such limit. Only a limit if it's same physical network.

For Live events on the road the usb tethering with mobile phones it's a salvation... there is really no way to solve the problem that iphones and android devices are using all the same lan subnet?

[Ysurac]

As I don't know of such problem, I don't know.

[openaspace]

As I don't know of such problem, I don't know.

The solution could be to isolate each device within a dedicated different subnet.

For each device the system will create a different subnet where to receive the device subnet, in this way each device net will be managed and isolated from others.

[Mutanter]

Ok, bear in mind and just to clarify : If you connect your tethered Phone, you'll get a new ETH interface in OMR / OpenWRT. For example: new Interface for your phone is ETH2 If you disconnect this Phone your ETH2 Interface will be "deleted" but it is still visible in OMR Status page. Suppose we now connect a new interface (either a USB LTE stick or another cell phone or another USB to LAN interface etc.). ) is automatically assigned to the device ETH 2 and this is exactly what causes problems. OMR still "hold" the "old" ETH2 configuration while OpenWRT / linux Kernel re-assigned ETH2 to the "new" device.

[openaspace]

Really thank you. I'm understanding that will be better to buy 4 LTE router , each one on different subnet and connect each one to the router bonding by ethernet cable.... it's only sane way to operate...

In a live environment, where i need to change live the sim cards, and detaching and reattaching the usb dongle, the configuration can change randomly

https://www.amazon.it/TP-Link-TL-MR6400-Wireless-configurazione-Staccabili/dp/B07RM95YFC/ref=sxin_9?__mk_it_IT=%C3%85M%C3%85%C5%BD%C3%95%C3%91&ascsubtag=amzn1.osa.50162fc4-bfd5-4799-9e98-21c50c667680.APJ6JRA9NG5V4.it_IT&creativeASIN=B07RM95YFC&cv_ct_cx=lte+router&cv_ct_id=amzn1.osa.50162fc4-bfd5-4799-9e98-21c50c667680.APJ6JRA9NG5V4.it_IT&cv_ct_pg=search&cv_ct_wn=osp-single-source-gl-ranking&dchild=1&keywords=lte+router&linkCode=oas&pd_rd_i=B07RM95YFC&pd_rd_r=adb6a37b-d481-4e25-9d12-b66fe7c0aced&pd_rd_w=956wi&pd_rd_wg=f2stX&pf_rd_p=682cfa5b-bd1b-4489-9db5-a06e4df71588&pf_rd_r=KEVQJA1DHCA49NE308MT&qid=1601903359&sr=1-1-5b72de9d-29e4-4d53-b588-61ea05f598f4&tag=aranzullait-osp-21

[Malaga82]

probably the best you can buy with budget is the Huawei B535, at least is 2CA.

[openaspace]

ok, but I had no idea that this was tethering and you said gateway was not the same. You can still try with only 2 connections (only one tethering) and check if same issue. Also set one connection as master as I said.

After the last the system was used and working perfectly, now without change nothing, with 1 LTE tethering and 1 Lan device, the redundancy restart only on the tether device and not on the lan , after a disconnection...

.....

[openaspace]

The problem is always the same.. 1) both wan are online 2) I start the streaming and both wan I see are transferring data at the same speed 3) detach cable from wan1 and the stream works over wan 2 4) reattach wan 1 and the stream continue to be transfered only over wan 2

If I stop the TCP (streaming) connection and restart it, the redundancy restart on both wan!

This replicate wan1 to wan2. GENERAL: https://www.dropbox.com/s/jofw77oz2is12vm/OpenMPTCProuter-Stato-LuCI.png?dl=0 MPTCP: https://www.dropbox.com/s/x4kpzb2dnp3yeqn/mptcp.png?dl=0

Here a usb device detached and reattached... with no redundancy start... the same happen without detach the usb cable and disabling only the data. The same happen on a normal LAN cable.

Wed Oct 7 17:21:13 2020 kern.info kernel: [ 1626.108951] usb 2-3: USB disconnect, device number 16 Wed Oct 7 17:21:13 2020 kern.info kernel: [ 1626.631561] usb 2-3: new high-speed USB device number 17 using ehci-pci Wed Oct 7 17:21:13 2020 daemon.notice netifd: Interface 'wan2' is enabled Wed Oct 7 17:21:13 2020 daemon.notice netifd: Interface 'wan2' is setting up now Wed Oct 7 17:21:13 2020 daemon.notice netifd: Network device 'usb0' link is up Wed Oct 7 17:21:13 2020 daemon.notice netifd: Interface 'wan2' has link connectivity Wed Oct 7 17:21:13 2020 kern.info kernel: [ 1626.790964] rndis_host 2-3:1.0 usb0: register 'rndis_host' at usb-0000:00:13.2-3, RNDIS device, 5a:3c:05:d4:b8:68 Wed Oct 7 17:21:13 2020 daemon.notice netifd: wan2 (26993): udhcpc: started, v1.31.1 Wed Oct 7 17:21:13 2020 daemon.notice netifd: wan2 (26993): udhcpc: sending discover Wed Oct 7 17:21:14 2020 user.notice ModemManager: hotplug: add network interface usb0: event processed Wed Oct 7 17:21:14 2020 user.notice ModemManager: hotplug: event reported: action=add, name=usb0, subsystem=net Wed Oct 7 17:21:14 2020 user.notice ModemManager: hotplug: no need to wait for modem at sysfs path /sys/devices/pci0000:00/0000:00:13.2/usb2/2-3 Wed Oct 7 17:21:15 2020 daemon.info vnstatd[1422]: Info: Interface "usb0" enabled. Wed Oct 7 17:21:16 2020 daemon.info [5950]: [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:13.2/usb2/2-3': not supported by any plugin Wed Oct 7 17:21:16 2020 daemon.notice netifd: wan2 (26993): udhcpc: sending discover Wed Oct 7 17:21:16 2020 daemon.notice netifd: wan2 (26993): udhcpc: sending select for 192.168.42.117 Wed Oct 7 17:21:16 2020 daemon.notice netifd: wan2 (26993): udhcpc: lease of 192.168.42.117 obtained, lease time 3600 Wed Oct 7 17:21:16 2020 daemon.debug dnsmasq[26321]: listening on usb0(#59): 192.168.42.117 port 53 Wed Oct 7 17:21:16 2020 daemon.notice netifd: Interface 'wan2' is now up Wed Oct 7 17:21:16 2020 user.notice firewall: Reloading firewall due to ifup of wan2 (usb0) Wed Oct 7 17:21:17 2020 user.notice mptcp: Reloading mptcp config due to ifup of wan2 (usb0) Wed Oct 7 17:21:19 2020 user.notice omr-bypass: Starting OMR-ByPass... Wed Oct 7 17:21:20 2020 user.notice post-tracking-post-tracking: Set server vps (xxx.xxx.xxx.xxx) default route xxx.xxx.xxx.xxx nexthop via 192.168.42.129 dev usb0 weight 1 nexthop via 192.168.20.1 dev eth1 weight 10 Wed Oct 7 17:21:20 2020 user.notice post-tracking-post-tracking: wan2 (usb0) switched up Wed Oct 7 17:21:20 2020 user.notice omr-bypass: Restart dnsmasq... Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[26321]: exiting on receipt of SIGTERM Wed Oct 7 17:21:20 2020 user.notice dnsmasq: DNS rebinding protection is active, will discard upstream RFC1918 responses! Wed Oct 7 17:21:20 2020 user.notice dnsmasq: Allowing 127.0.0.0/8 responses Wed Oct 7 17:21:20 2020 user.notice dnsmasq: Allowing RFC1918 responses for domain plex.direct Wed Oct 7 17:21:20 2020 user.notice omr-bypass: OMR-ByPass is running Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: started, version 2.82 cachesize 150 Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: DNS service limited to local subnets Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: compile time options: IPv6 GNU-getopt no-DBus UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP conntrack ipset auth DNSSEC no-ID loop-detect inotify dumpfile Wed Oct 7 17:21:20 2020 daemon.info dnsmasq-dhcp[29285]: DHCP, IP range 192.168.100.100 -- 192.168.100.249, lease time 12h Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using only locally-known addresses for domain test Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using only locally-known addresses for domain onion Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using only locally-known addresses for domain localhost Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using only locally-known addresses for domain local Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using only locally-known addresses for domain invalid Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using only locally-known addresses for domain bind Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using only locally-known addresses for domain use-application-dns.net Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using only locally-known addresses for domain lan Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using nameserver 127.0.0.1#5353 Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: using 1 more local addresses Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: read /etc/hosts - 4 addresses Wed Oct 7 17:21:20 2020 daemon.info dnsmasq[29285]: read /tmp/hosts/dhcp.cfg01411c - 1 addresses Wed Oct 7 17:21:20 2020 daemon.info dnsmasq-dhcp[29285]: read /etc/ethers - 0 addresses

[Ysurac]

I'm not able to reproduce the issue. Can you try to use omr-test-speed and do a plug off/plug on ? And check speed and MPTCP realtime traffic ? i'm really not sure that MPTCP redundant mode really work well in all case...

[openaspace]

with MPTCP enabled: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 9 10.0G 9 984M 0 0 11.6M 0 0:14:39 0:01:24 0:13:15 12.7M^Z[8]+ Stopped

without MPTCP enabled: % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 8 10.0G 8 826M 0 0 6773k 0 0:25:48 0:02:04 0:23:44 6751k^Z[9]+ Stopped

[openaspace]

i'm really not sure that MPTCP redundant mode really work well in all case...

It's the only mode that I can have for redundancy, with "default" or "round-robin" when a WAN go off there are some seconds offline.. and after start on the other WAN.

For my test the only modality is the "redundancy" .. having a continuous data transfer. This problem don't exit in normal traffic where download and upload are casually .. and for office internet use it's a perfect redundancy.. but not for live applications.

[Ysurac]

It's not what I asked... Detach and attach device during an omr-test-speed and check speed and graph.

Default shouldn't take offline for seconds. Live applications should have a small cache in most case.

[openaspace]

Detach and attach device during an omr-test-speed and check speed and graph.

Yes, If I disconnect one WAN during the omr test I see the test blocked for some seconds and after restart.

[Ysurac]

blocked ? It may slow but shouldn't block. On the graph you see all wan at 0 for some seconds ? How many seconds ?

[openaspace]

Live applications should have a small cache in most case. there is no buffer or cache server side, the buffer is client side where the ffmpeg encoder works, but is for anticipate random problem on the client, not on the server (but in any case for disconnection problems.. and using professional hardware encoders there is not this kind of interactivity.. you will see the stream going off and stop... using instead a software there are more opportunity for the software to interact with connection problems in a fast way, the hardware is slow and intended to be used in a solid configuration environment and are used for the best video quality - an ffmpeg process can use 50% of a i7 cpu, the hardware encoder go all with small resources and best video quality). Only with nginx and HLS (where there is not a buffer the same) the problem it's not really evident for the packet impulse every 6 seconds to the end visitors and because works over tcp. But for example with vimeo .. google cloud etc.. the visitors will see black screen.. and in the bad prevision the cloud server will go in panic.. rejecting to start the server transcoder if a reconnection occur in sub or 1 second! and in a CDN vision with thousands of visitors...it's the panic (this transcoding problem don't apply only on nginx with hls)

[openaspace]

blocked ? It may slow but shouldn't block. On the graph you see all wan at 0 for some seconds ? How many seconds ?

don't go to 0 , remain blocked at the last value for 2/3 seconds.

[openaspace]

yes now was go to 0 and never restart when connected newly the disconnected wan

[openaspace]

could be hardware problem? could be that using dedicated hardware server all this will not happen?

I can try to switch to a DELL T310 xeon server...

[Ysurac]

I don't think but you can try....

It's strange that even with omr-test-speed connection doesn't restart. This happen also with omr-test-speed with something that is not using usb ? (in this case it can be a slow initialization of usb that need to be fixed by a delay).

You can also check memory usage and load average on the router ?

[openaspace]

Video : https://www.dropbox.com/s/4i9lulr1uo8fhie/v2Redundant.mp4?dl=0

How you can see the offline wan that is returned on don't restart redundancy. If I stop the stream and restart , the redundancy restart.

[openaspace]

I don't think but you can try....

It's strange that even with omr-test-speed connection doesn't restart. This happen also with omr-test-speed with something that is not using usb ? (in this case it can be a slow initialization of usb that need to be fixed by a delay).

You can also check memory usage and load average on the router ?

the cpu go to 80/90% , ram 220 used over 3,6G available

[openaspace]

OMR speed test video: https://www.dropbox.com/s/b6h8cle2okwzwj5/omr%20speed.mov?dl=0

you can see the speed going down when disconnect one wan. apparently no disconnection, but OBS and VMIX was going off at the disconnection moment... but i think this was for the 90% cpu load problem.

could be that using dedicated hardware server , this possible latency could be reduced ..

[Ysurac]

The 90% CPU load is a problem. With htop on the router you can see if it's a program that use so much CPU.

[openaspace]

v2ray and ss-redir are using all the cpu during the omr test.

anyway .. working on low bandwidth (3,5mbps) the redundancy works on the first step, 2 wan on, one wan go off and the streaming continue to works correctly over the wan 2.. and this is a partial success.. because exist a backup, but don't restart the wan 1.. and all this happen with cpu at 22% load..

[Ysurac]

v2ray ? why v2ray is used ? you enabled obfuscation ?

[openaspace]

v2ray ? why v2ray is used ? you enabled obfuscation ?

yes, but the problem was the same also without obfuscation

[openaspace]

without v2ray cpu at 50% but no changes on the redundant problem

[openaspace]

The 90% CPU load is a problem. With htop on the router you can see if it's a program that use so much CPU.

So.. I moved my configuration on the DELL T310 8 core Xeon, using the 2 embed NIC, the cpu is at 2% when transferring at 100mbps, ram used 165M on 64GB available.

1. The problem of redundancy that not restart replicate;
2. I disconnect and reconnect a Wan and don't restore the redundancy when available; Restarting the tcp connection, the redundancy restart;
3. strange speed test, i have fiber at 1 gbps down and 250mbps up, the remote server have 1gbps in/out guaranteed unmetered unfiltered, but behind the router it reach only 120mbps in down and 87 in up (both nic are 1gbps) test repeated more times. Could be that is because DELL T310 don't support AES-NI?

Last test tomorrow with a Huawei LTE PRO router that will avoid me to use USB dongle and tethering.....giving me directly a lan gateway.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6387389/

[openaspace]

UPDATE PRO: Changing the Congestion Control to wvegas the number of restored redundant connection is really high , but no happen every time, but really better. CON: instable connection on single wan

[openaspace]

The day after the wvegas not works as yesterday....

I have changed lte connection on a professional router .. and the result is the same.. no redundant connection start after a disconnection/reconnection...

At this point there is something in the mptcp kernel for redundant settings that doesn't works around the existing tcp connections..

Where i can try to look in the kernel configuration? Could be also a problem in the remote VPS server?

[openaspace]

Could be , to manually set static routes of each LTE router?

[openaspace]

Expected Behavior When a WAN go offline and come back online , the redundancy will restart on each wan device

Current Behavior I have 2 Lan WAN (physical ethernet devices) + 1 Wan using tethered phones. When I start a new tcp connection the redundancy start correctly on all wan, but disconnecting and reconnecting 1 wan device the redundancy don't restart. Also happen all the same, that with 2 wan on and 1 wan off, that disconnecting and reconnecting one of the 2 wan on, the 3rd with redundancy not started previously, start !

Possible Solution Possible routing problem at reconnection of a previous off wan device.

Steps to Reproduce the Problem 3 WAN connected, tunnel working, redundancy working. disconnect and reconnect WAN 1 and redundancy don't start newly on WAN 1 I disconnect WAN 2 (redundancy on wan 1 remain off) I reconnect WAN 2 No redundancy restart on WAN 2 but redundancy on WAN 1 start! Redundancy on wan 2 remain off. Context (Environment) I use mptcp router in redundancy mode. I have'd the same problem previously, therefore i was started from new installation and 0 configuration Also tried to change from laptop with lan usb adapter to a dedicated dell server with 3 embed nic with the same problem, that replicate exactly

Specifications OpenMPTCProuter version: v0.55.3 r0+14198-da98603597 OpenMPTCProuter VPS version: OpenMPTCProuter platform: AMD 64bit single core

[openaspace]

The same happen with "round-robin" with the only difference that the active tcp connection go down from the client.

With only 2 wan, the redundancy and round robin are unusable. I will try to check with 4 wan in the next days... and if the 3 wan dynamic, will replicate exactly.. because if replicate always in the same mode, could be a partially secure redundancy ....

[openaspace]

2 december, new setup from 0

same problems, with 2 wan connection, no failover but with 3 wan the failover works..

[Ysurac]

In redundant mode ?

[openaspace]

In redundant mode ?

Redundant or not, the problem is the same. With 3 wan in redundant it works better but it's not stable, when reconnecting a wan, the switch between the WANs create small time-out and when a switch occur, the initial ping is high. Also when there is a wan switch, the connected lan client sometimes employ some seconds to obtain connectivity...

Hardware i5,15gb ddr3,embed ethernet + plus 2 pci ethernet + 2 usb ethernet

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days

[lferreira93]

@openaspace Do you have any evolution in your problem? I want do the same, streaming via SRT with high avaliability, using 4 4G USb modens.

[openaspace]

I'm not using anymore this system.. But when I was obligated to find a solution time ago... I have worked with one connection at the time... Without automatic switch..I mean.. That in the case of down of one connection, I manually changes the gateway of the computer with vmix, without this system. Not a top solution yes.

2 or more LTE connections, manual network IP on the device, and on the case of failure I was ready to switch the gateway to the second one LTE connection.

For other mission critical jobs, I used peplink devices. You can rent from local distributors.

[Llorx]

@openaspace How is Peplink working? We got a quote that is a bit over our budget, but if is reliable, then we may go for it. Did you notice a bandwidth penalty on unreliable connections? I mean, if you have 100mbit starlink and a 4G connection that is dropping suddenly to 0.5mbit, do the bonded connection suffer and drop to 50mbit or whatever? That happened to me with Speedify so I'm a bit reluctant.