Can't get glorytun to use multiple paths

izomiac commented 6 years ago

Hopefully I've just don't something stupid here, but I can't get glorytun to use multiple outbound interfaces. Following the readme I get to where ping works fine, but when I add a second path glorytun creates two connections using a single interface rather than having each connection use the specified interface. If I take down the original interface but leave the other up then glorytun won't failover and packets get dropped. I also don't think it matters which order I add the interfaces in, I think it just uses the one with the lower metric in the routing table.

I'm using a Debian VPS acting as server and a Linksys WRT3200ACM (OpenWrt 18.06.1, glorytun version 4b4a6308e711e6fb614d4886a9887fac040a1d56) acting as client. 192.168.10.2 is an ethernet connection and 192.168.1.249 is an 802.11n client.

root@Mahoro:~# glorytun-udp path dev tun0
path UP
  bind:     192.168.10.2 port 5000
  public:   162.233.10.148 port 5000
  peer:     [Redacted server IP] port 5000
  mtu:      1472 bytes
  rtt:      118.300 ms
  rttvar:   28.076 ms
  upload:   1005 bytes/s (max: 2094)
  download: 1012 bytes/s (max: 3458)
  output:   1337 packets
  input:    603 packets
path UP
  bind:     192.168.1.249 port 5000
  public:   162.233.10.148 port 1024
  peer:     [Redacted server IP] port 5000
  mtu:      1472 bytes
  rtt:      113.877 ms
  rttvar:   27.377 ms
  upload:   2098 bytes/s (max: 2770)
  download: 131 bytes/s (max: 2079)
  output:   1300 packets
  input:    934 packets

Both paths are reported as having the same public IP address, despite being from different ISPs, suggesting two connections from the same interface.

angt commented 6 years ago

Hi! First, thank you for taking the time to write this issue :) Multipath setup is hard, first you need to check that your routes are ok. For example, on my device I can check this with:

root@OverTheBox:~# ip route get 8.8.8.8 from 10.0.0.104
8.8.8.8 from 10.0.0.104 via 10.0.0.1 dev eth2 table 201 uid 0 
    cache 
root@OverTheBox:~# ip route get 8.8.8.8 from 10.0.1.119
8.8.8.8 from 10.0.1.119 via 10.0.1.1 dev eth1 table 200 uid 0 
    cache

Each path should have its own table and gw. Also, glorytun is still under dev and you should wait the 1.0 for production.

izomiac commented 6 years ago

Well, that did turn out to be an issue, but apparently not the only issue.

root@Mahoro:~# ip route get [Redacted Server IP] from 192.168.1.249
[Redacted Server IP] from 192.168.1.249 via 192.168.1.1 dev wlan3 table wwan0 uid 0
    cache
root@Mahoro:~# ip route get [Redacted Server IP] from 192.168.10.2
[Redacted Server IP] from 192.168.10.2 via 192.168.10.1 dev eth1.2 uid 0
    cache expires 502sec mtu 1428
root@Mahoro:~# glorytun-udp path dev tun0
path UP
  bind:     192.168.10.2 port 5000
  public:   166.137.96.227 port 30193
  peer:     [Redacted Server IP] port 5000
  mtu:      1400 bytes
  rtt:      391.812 ms
  rttvar:   203.270 ms
  upload:   913 bytes/s (max: 7397)
  download: 936 bytes/s (max: 2265)
  output:   783 packets
  input:    624 packets
path UP
  bind:     192.168.1.249 port 5000
  public:   166.137.96.227 port 46800
  peer:     [Redacted Server IP] port 5000
  mtu:      1391 bytes
  rtt:      477.718 ms
  rttvar:   90.019 ms
  upload:   306 bytes/s (max: 8627)
  download: 305 bytes/s (max: 2069)
  output:   187 packets
  input:    70 packets
root@Mahoro:~#

But at least that gives me something to work with, which is a lot more than I had yesterday. It's probably still a routing issue.

Thankfully, this isn't in a production environment. Just a home user who discovered the two local ISPs won't take any new customers, so I'm trying to aggregate ~4 long-range connections to WiFI access points and a cellular internet hotspot to get something approaching a reliable internet connection. Glorytun is actually one of the more mature projects for this, unless I want to use a Windows-based application or enterprise-level hardware.

angt / glorytun

Can't get glorytun to use multiple paths #28