search

Ysurac / openmptcprouter

OpenMPTCProuter is an open source solution to aggregate multiple internet connections using Multipath TCP (MPTCP) on OpenWrt
https://www.openmptcprouter.com/
GNU General Public License v3.0
1.8k stars 258 forks source link

When shutdown VPS (crash or poweroff or ...) all WAN is down and can't get internet. #2388

Closed Kalimeiro closed 2 years ago

Kalimeiro commented 2 years ago

Expected Behavior

When VPS is down, if disable default gateway is not check, the master WAN will be used to direct output, then if master WAN is down OMR use another WAN up and running to direct output, and when VPS is up, the aggregate start on all WAN up and running.

Current Behavior

When VPS is down, no direct output selected, need to restart OMR to have the master WAN working.

Step to reproduce

Shutdown the VPS, and see status page.

Specifications

Kalimeiro commented 2 years ago

Before restart VPS : image

After restart VPS : image

Kalimeiro commented 2 years ago

i have found how to reproduce the problem, the VPS crash because of shorewall and a bug with port forwarding.

Kalimeiro commented 2 years ago

@Ysurac this bug is hard, if i shutdown the VPS i lost totaly internet, master WAN isn't used to direct output, i need to disable the server on OMR to get internet and start my VPS again and then reenable the server / OR / just restart the RPI.

In the same time the VPS is down, OMR see all WAN as down. / When OMR is restart all WAN is down, but internet on master WAN work more than 1 minute waiting.

Kalimeiro commented 2 years ago

Before shutdown the VPS

image

After shutdown the VPS

image

NOTE : i'm waiting few minutes, but no internet, so i decide to restart OMR / OR / you can disable the server.

When OMR is restarted

image

NOTE : i need to wait more than 1 minute to have internet on the master WAN.

start the VPS and waiting OMR to connect and aggregate

image

Ysurac commented 2 years ago

What is the proxy and VPN used ? What do you have in status->system log when you shutdown VPS ?

Kalimeiro commented 2 years ago

I use Shadowsocks and GloryTUN TCP

When shutdown VPS, systemlog show : 

Tue Jun 21 13:01:16 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:01:20 2022 daemon.info omr-tracker-ss: Shadowsocks sss0 is down (can't contact via http 198.41.212.162, 1.0.0.1, 212.27.48.10)
Tue Jun 21 13:01:20 2022 user.notice Shadowsocks: Rules DOWN
Tue Jun 21 13:01:32 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:01:38 2022 daemon.info omr-tracker-ss: Server sss0 (37.59.xxxxxxxxxxx) seems down, no answer to ping
Tue Jun 21 13:01:48 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:02:59 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:03:15 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:03:31 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:04:02 2022 user.notice post-tracking-post-tracking: omrvpn down because check error, interface have no IPv6, interface have no IPv6 gateway and ping from 10.255.255.2 error (4.2.2.1,8.8.8.8,4.2.2.1)
Tue Jun 21 13:04:02 2022 user.notice post-tracking-post-tracking: Glorytun VPN down, restart it
Tue Jun 21 13:04:02 2022 daemon.err glorytun[16230]: getpeername: Socket not connected
Tue Jun 21 13:04:02 2022 daemon.notice netifd: Network device 'tun0' link is down
Tue Jun 21 13:04:02 2022 daemon.notice netifd: Interface 'omrvpn' has link connectivity loss
Tue Jun 21 13:04:02 2022 daemon.notice netifd: Interface 'omrvpn' is now down
Tue Jun 21 13:04:02 2022 daemon.info glorytun: starting glorytun vpn instance vpn
Tue Jun 21 13:04:02 2022 daemon.notice netifd: Interface 'omrvpn' is disabled
Tue Jun 21 13:04:02 2022 daemon.info glorytun[8529]: INITIALIZED tun0
Tue Jun 21 13:04:02 2022 daemon.notice netifd: Interface 'omrvpn' is enabled
Tue Jun 21 13:04:02 2022 daemon.notice netifd: Network device 'tun0' link is up
Tue Jun 21 13:04:02 2022 daemon.notice netifd: Interface 'omrvpn' has link connectivity
Tue Jun 21 13:04:02 2022 daemon.notice netifd: Interface 'omrvpn' is setting up now
Tue Jun 21 13:04:02 2022 daemon.notice netifd: Interface 'omrvpn' is now up
Tue Jun 21 13:04:03 2022 user.notice firewall: Reloading firewall due to ifup of omrvpn (tun0)
Tue Jun 21 13:04:03 2022 user.notice Shadowsocks: Rules UP
Tue Jun 21 13:04:03 2022 user.notice Shadowsocks: Reload omr-bypass rules
Tue Jun 21 13:04:03 2022 user.notice omr-bypass: Starting OMR-ByPass...
Tue Jun 21 13:04:03 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:04:05 2022 user.notice omr-bypass: Reload dnsmasq...
Tue Jun 21 13:04:06 2022 daemon.info dnsmasq[1]: read /etc/hosts - 4 addresses
Tue Jun 21 13:04:06 2022 daemon.info dnsmasq[1]: read /tmp/hosts/dhcp.cfg01411c - 6 addresses
Tue Jun 21 13:04:06 2022 daemon.info dnsmasq-dhcp[1]: read /etc/ethers - 0 addresses
Tue Jun 21 13:04:06 2022 user.notice omr-bypass: OMR-ByPass is running
Tue Jun 21 13:04:06 2022 daemon.warn dnsmasq[1]: failed to create listening socket for 10.255.255.2: Address not available
Tue Jun 21 13:04:10 2022 daemon.info omr-tracker-ss: Shadowsocks sss0 is down (can't contact via http 1.0.0.1, 212.27.48.10, 198.27.92.1, 151.101.129.164, 77.88.55.77, 1.1.1.1, 74.82.42.42)
Tue Jun 21 13:04:10 2022 user.notice Shadowsocks: Rules DOWN
Tue Jun 21 13:04:19 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:04:26 2022 daemon.info omr-tracker-ss: Server sss0 (37.59.xxxxxxxxx) seems down, no answer to ping
Tue Jun 21 13:04:31 2022 user.notice post-tracking-post-tracking: Change in routes, set ip route replace default scope global  nexthop via 192.168.2.254 dev eth1.20 weight 2 nexthop via 192.168.3.254 dev eth1.10 weight 1 (omrvpn_intf: tun0)
Tue Jun 21 13:04:35 2022 user.notice mptcp: Reloading mptcp config due to ifup of omrvpn (tun0)
Tue Jun 21 13:04:37 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:04:53 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:05:10 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:05:26 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:05:42 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:05:58 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:06:14 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:06:31 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:06:47 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)
Tue Jun 21 13:07:03 2022 user.notice OMR-VPS: Can't get vps token, try later (can't ping server vps on 37.59.xxxxxxxxx, no server API answer on 37.59.xxxxxxxxx)

~2/3 minutes later when OMR detect GloryTun is down, he change the default route (rarely) and i get internet through Master WAN, OMR say direct output with the IP PUBLIC of my Freebox, but all wan have the message "no server ping response after 1 second".

Most of times no direct output and no internet, i need to restart the RPI to have direct output, or disable the server in the wizard, or just wait.

EDIT : on another OMR client, master WAN (fibre orange) is not selected to direct output but another WAN (ADSL Orange which is the worst connection with low bandwith)

Kalimeiro commented 2 years ago

For all, if you experience this similar problem, you need to disable TC (TC_ENABLED=No) from Shorewall config (/etc/shorewall/shorewall.conf) in your VPS.