search

sudomesh / sudowrt-firmware

Scripts to build the sudo mesh OpenWRT firmware.
Other
73 stars 19 forks source link

ssh not working over homenodes #63

Closed max-b closed 9 years ago

max-b commented 9 years ago

I'll attach a pcap of me, wireless connected to the 'open2' interface ("peoplesopen.net"), doing a bit of web browsing successfully and then failing to be able to ssh to any servers, both mesh and off the mesh.

Here are the configs from a TP-link wdr4300 which is on the most current firmware/makenode code (as of 7/27/2015), and then below I'll post a set of configs from another of the same device that's running some older code that does seem to be working:

root@my:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether c4:6e:1f:b9:f0:6c brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c66e:1fff:feb9:f06c/64 scope link 
       valid_lft forever preferred_lft forever
3: teql0: <NOARP> mtu 1500 qdisc noop state DOWN group default qlen 100
    link/void 
20: br-open: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c4:6e:1f:b9:f0:6c brd ff:ff:ff:ff:ff:ff
    inet 100.64.4.1/26 brd 100.64.4.63 scope global br-open
       valid_lft forever preferred_lft forever
    inet6 fe80::c66e:1fff:feb9:f06c/64 scope link 
       valid_lft forever preferred_lft forever
21: eth0.10@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-open state UP group default 
    link/ether c4:6e:1f:b9:f0:6c brd ff:ff:ff:ff:ff:ff
22: br-priv: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c4:6e:1f:b9:f0:6c brd ff:ff:ff:ff:ff:ff
    inet 172.30.0.1/16 brd 172.30.255.255 scope global br-priv
       valid_lft forever preferred_lft forever
    inet6 fe80::c66e:1fff:feb9:f06c/64 scope link 
       valid_lft forever preferred_lft forever
23: eth0.11@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-priv state UP group default 
    link/ether c4:6e:1f:b9:f0:6c brd ff:ff:ff:ff:ff:ff
24: eth0.2@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c4:6e:1f:b9:f0:6c brd ff:ff:ff:ff:ff:ff
    inet 100.64.4.1/32 brd 255.255.255.255 scope global eth0.2
       valid_lft forever preferred_lft forever
    inet6 fe80::c66e:1fff:feb9:f06c/64 scope link 
       valid_lft forever preferred_lft forever
25: eth0.3@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c4:6e:1f:b9:f0:6c brd ff:ff:ff:ff:ff:ff
    inet 100.64.4.1/32 brd 255.255.255.255 scope global eth0.3
       valid_lft forever preferred_lft forever
    inet6 fe80::c66e:1fff:feb9:f06c/64 scope link 
       valid_lft forever preferred_lft forever
26: eth0.1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether c4:6e:1f:b9:f0:6c brd ff:ff:ff:ff:ff:ff
    inet 172.22.0.1/24 brd 172.22.0.255 scope global eth0.1
       valid_lft forever preferred_lft forever
    inet 192.168.0.154/24 brd 192.168.0.255 scope global eth0.1
       valid_lft forever preferred_lft forever
    inet6 fd33:801d:c091:0:c66e:1fff:feb9:f06c/64 scope global dynamic 
       valid_lft 6698sec preferred_lft 1298sec
    inet6 2601:643:8200:baa3:c66e:1fff:feb9:f06c/64 scope global dynamic 
       valid_lft 6698sec preferred_lft 1298sec
    inet6 fd33:801d:c091::529/128 scope global dynamic 
       valid_lft 85103sec preferred_lft 85103sec
    inet6 2601:643:8200:baa3::529/128 scope global dynamic 
       valid_lft 85103sec preferred_lft 85103sec
    inet6 fe80::c66e:1fff:feb9:f06c/64 scope link 
       valid_lft forever preferred_lft forever
27: mesh5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether c4:6e:1f:b9:f0:6e brd ff:ff:ff:ff:ff:ff
    inet 100.64.4.1/32 brd 255.255.255.255 scope global mesh5
       valid_lft forever preferred_lft forever
    inet6 fe80::c66e:1fff:feb9:f06e/64 scope link 
       valid_lft forever preferred_lft forever
28: mesh2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether c4:6e:1f:b9:f0:6d brd ff:ff:ff:ff:ff:ff
    inet 100.64.4.1/32 brd 255.255.255.255 scope global mesh2
       valid_lft forever preferred_lft forever
    inet6 fe80::c66e:1fff:feb9:f06d/64 scope link 
       valid_lft forever preferred_lft forever
29: open5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-open state UP group default qlen 1000
    link/ether c6:6e:1f:b9:f0:6e brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c46e:1fff:feb9:f06e/64 scope link 
       valid_lft forever preferred_lft forever
30: open2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-open state UP group default qlen 1000
    link/ether c6:6e:1f:b9:f0:6d brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c46e:1fff:feb9:f06d/64 scope link 
       valid_lft forever preferred_lft forever
31: priv2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-priv state UP group default qlen 1000
    link/ether c2:6e:1f:b9:f0:6d brd ff:ff:ff:ff:ff:ff
32: priv5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br-priv state UP group default qlen 1000
    link/ether c2:6e:1f:b9:f0:6e brd ff:ff:ff:ff:ff:ff
33: l2tp0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1446 qdisc htb state UNKNOWN group default qlen 1000
    link/ether c2:85:9b:67:05:c0 brd ff:ff:ff:ff:ff:ff
    inet 100.64.4.1/32 scope global l2tp0
       valid_lft forever preferred_lft forever
    inet6 fe80::c085:9bff:fe67:5c0/64 scope link 
       valid_lft forever preferred_lft forever
root@my:~# ip route
default via 192.168.0.1 dev eth0.1  proto static 
100.64.4.0/26 dev br-open  proto kernel  scope link  src 100.64.4.1 
172.22.0.0/24 dev eth0.1  proto kernel  scope link  src 172.22.0.1 
172.30.0.0/16 dev br-priv  proto kernel  scope link  src 172.30.0.1 
192.168.0.0/24 dev eth0.1  proto kernel  scope link  src 192.168.0.154 
root@my:~# ip route show table public
default via 100.64.0.1 dev l2tp0  proto 42 onlink 
100.64.0.1 via 100.64.0.1 dev l2tp0  proto 42 onlink 
100.64.0.10 via 100.64.0.1 dev l2tp0  proto 42 onlink 
100.64.2.64/26 via 100.64.0.1 dev l2tp0  proto 42 onlink 
100.64.2.66 via 100.64.0.1 dev l2tp0  proto 42 onlink 
100.64.2.192/26 via 100.64.2.193 dev mesh5  proto 42 onlink 
100.64.3.64/26 via 100.64.0.1 dev l2tp0  proto 42 onlink 
100.64.4.0/26 dev br-open  proto kernel  scope link  src 100.64.4.1 
172.22.0.2 via 100.64.0.1 dev l2tp0  proto 42 onlink 
root@my:~# iptables-save
# Generated by iptables-save v1.4.21 on Fri Jul 31 11:15:24 2015
*nat
:PREROUTING ACCEPT [422:41695]
:INPUT ACCEPT [96:7775]
:OUTPUT ACCEPT [7:4758]
:POSTROUTING ACCEPT [257:24935]
-A POSTROUTING -o eth0.1 -j MASQUERADE
-A POSTROUTING -s 172.30.0.0/24 -d 100.64.0.0/10 -j MASQUERADE
COMMIT
# Completed on Fri Jul 31 11:15:24 2015
# Generated by iptables-save v1.4.21 on Fri Jul 31 11:15:24 2015
*raw
:PREROUTING ACCEPT [88011:25527159]
:OUTPUT ACCEPT [45070:20405598]
COMMIT
# Completed on Fri Jul 31 11:15:24 2015
# Generated by iptables-save v1.4.21 on Fri Jul 31 11:15:24 2015
*mangle
:PREROUTING ACCEPT [514:62779]
:INPUT ACCEPT [298:31145]
:FORWARD ACCEPT [216:31634]
:OUTPUT ACCEPT [268:51111]
:POSTROUTING ACCEPT [484:82745]
-A FORWARD -o l2tp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
-A FORWARD -i l2tp0 -p tcp -m tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
COMMIT
# Completed on Fri Jul 31 11:15:24 2015
# Generated by iptables-save v1.4.21 on Fri Jul 31 11:15:24 2015
*filter
:INPUT DROP [115:15860]
:FORWARD DROP [1:40]
:OUTPUT ACCEPT [11464:9336834]
-A INPUT -s 100.64.0.0/10 -d 172.30.0.0/24 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -s 100.64.0.0/10 -d 172.30.0.0/24 -j DROP
-A INPUT -i lo -j ACCEPT
-A INPUT -s 100.64.0.0/10 -i mesh2 -j ACCEPT
-A INPUT -s 100.64.0.0/10 -i mesh5 -j ACCEPT
-A INPUT -s 100.64.0.0/10 -i l2tp0 -j ACCEPT
-A INPUT -s 172.30.0.0/24 -i br-priv -j ACCEPT
-A INPUT -s 100.64.0.0/10 -i br-open -j ACCEPT
-A INPUT -s 100.64.0.0/10 -i eth0.2 -j ACCEPT
-A INPUT -s 100.64.0.0/10 -i eth0.3 -j ACCEPT
-A INPUT -s 100.64.0.0/10 -i eth0.4 -j ACCEPT
-A INPUT -s 100.64.0.0/10 -i eth0.5 -j ACCEPT
-A INPUT -i br-priv -p udp -m udp --sport 67:68 --dport 67:68 -j ACCEPT
-A INPUT -i br-open -p udp -m udp --sport 67:68 --dport 67:68 -j ACCEPT
-A INPUT -i eth0.1 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i mesh2 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i mesh5 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -i l2tp0 -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -d 255.255.255.255/32 -i eth0.2 -p udp -m udp --sport 4343 --dport 4242 -j ACCEPT
-A INPUT -d 255.255.255.255/32 -i eth0.3 -p udp -m udp --sport 4343 --dport 4242 -j ACCEPT
-A INPUT -d 255.255.255.255/32 -i eth0.4 -p udp -m udp --sport 4343 --dport 4242 -j ACCEPT
-A INPUT -d 255.255.255.255/32 -i eth0.5 -p udp -m udp --sport 4343 --dport 4242 -j ACCEPT
-A INPUT -s 172.22.0.0/24 -i eth0.1 -j ACCEPT
-A FORWARD -s 100.64.0.0/10 ! -d 172.30.0.0/24 -i mesh2 -j ACCEPT
-A FORWARD -s 100.64.0.0/10 ! -d 172.30.0.0/24 -i mesh5 -j ACCEPT
-A FORWARD -s 100.64.0.0/10 ! -d 172.30.0.0/24 -i l2tp0 -j ACCEPT
-A FORWARD -s 100.64.0.0/10 ! -d 172.30.0.0/24 -i br-open -j ACCEPT
-A FORWARD -s 100.64.0.0/10 ! -d 172.30.0.0/24 -i eth0.2 -j ACCEPT
-A FORWARD -s 100.64.0.0/10 ! -d 172.30.0.0/24 -i eth0.3 -j ACCEPT
-A FORWARD -s 100.64.0.0/10 ! -d 172.30.0.0/24 -i eth0.4 -j ACCEPT
-A FORWARD -s 100.64.0.0/10 ! -d 172.30.0.0/24 -i eth0.5 -j ACCEPT
-A FORWARD -s 172.30.0.0/24 -i br-priv -j ACCEPT
-A FORWARD -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -i l2tp0 -o br-priv -j DROP
-A FORWARD -i br-priv -o l2tp0 -j DROP
-A FORWARD -i mesh2 -o eth0.1 -j DROP
-A FORWARD -i mesh5 -o eth0.1 -j DROP
-A FORWARD -i l2tp0 -o eth0.1 -j DROP
-A FORWARD -i br-open -o eth0.1 -j DROP
-A FORWARD -i eth0.2 -o eth0.1 -j DROP
-A FORWARD -i eth0.3 -o eth0.1 -j DROP
-A FORWARD -i eth0.4 -o eth0.1 -j DROP
-A FORWARD -i eth0.5 -o eth0.1 -j DROP
-A FORWARD -i eth0.1 -o eth0.1 -j DROP
COMMIT
# Completed on Fri Jul 31 11:15:24 2015

kill -USR1 $(pgrep babeld); tail -n 100 /var/log/babeld.log

My id c6:6e:1f:ff:fe:b9:f0:6d seqno 38888
Neighbour fe80::290:a9ff:fe0d:87a3 dev mesh5 reach ff00 rxcost 258 txcost 258 rtt 0.000 rttcost 0 chan 157.
Neighbour fe80::c9f:5eff:fe6d:4b86 dev l2tp0 reach ff80 rxcost 96 txcost 96 rtt 0.000 rttcost 0 chan -2.
100.64.4.0/26 metric 128 (exported)
0.0.0.0/0 metric 96 (2368) refmetric 0 id 06:01:40:ff:fe:fc:42:01 seqno 64289 age 5 via l2tp0 neigh fe80::c9f:5eff:fe6d:4b86 nexthop 100.64.0.1 (installed)
100.64.0.1/32 metric 96 (1319) refmetric 0 id 06:01:40:ff:fe:fc:42:01 seqno 64289 age 5 via l2tp0 neigh fe80::c9f:5eff:fe6d:4b86 nexthop 100.64.0.1 (installed)
100.64.0.10/32 metric 192 (1253) refmetric 96 id 06:01:29:ff:fe:19:66:01 seqno 43195 age 5 via l2tp0 neigh fe80::c9f:5eff:fe6d:4b86 nexthop 100.64.0.1 (installed)
100.64.2.64/26 metric 320 (2391) refmetric 224 id 16:cc:20:ff:fe:b5:5c:24 seqno 13516 age 5 via l2tp0 neigh fe80::c9f:5eff:fe6d:4b86 nexthop 100.64.0.1 (installed)
100.64.2.66/32 metric 288 (2369) refmetric 192 id 26:a4:3c:ff:fe:bd:29:c7 seqno 57730 age 5 via l2tp0 neigh fe80::c9f:5eff:fe6d:4b86 nexthop 100.64.0.1 (installed)
100.64.2.192/26 metric 394 (6137) refmetric 128 id 02:90:a9:ff:fe:0d:87:a3 seqno 38236 age 10 via mesh5 neigh fe80::290:a9ff:fe0d:87a3 nexthop 100.64.2.193 (installed)
100.64.3.64/26 metric 320 (1729) refmetric 224 id 16:cc:20:ff:fe:75:c4:07 seqno 8337 age 5 via l2tp0 neigh fe80::c9f:5eff:fe6d:4b86 nexthop 100.64.0.1 (installed)
172.22.0.2/32 metric 288 (1416) refmetric 192 id 26:a4:3c:ff:fe:bd:29:c7 seqno 57730 age 5 via l2tp0 neigh fe80::c9f:5eff:fe6d:4b86 nexthop 100.64.0.1 (installed)
root@my:~# lsof -n -i TCP
COMMAND   PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
dropbear  878   root    4u  inet   1265      0t0  TCP *:ssh (LISTEN)
uhttpd   1366   root    3u  inet   2111      0t0  TCP 172.30.0.1:www (LISTEN)
uhttpd   1366   root    4u  inet   2112      0t0  TCP 172.30.0.1:https (LISTEN)
dnsmasq  1588 nobody    7u  inet   2648      0t0  TCP *:domain (LISTEN)
polipo   2002   root    3u  inet   3384      0t0  TCP *:8123 (LISTEN)
dropbear 2345   root    7u  inet   6099      0t0  TCP 100.64.4.1:ssh->100.64.4.53:57374 (ESTABLISHED)

A couple weird things:

max-b commented 9 years ago

Ok well I have good news and I have bad news.....

The good news is that I think I figured out what's causing this.

The bad news is that once again, it seems to be mtu issues :/ The home node had an mtu of 1446, while the exit server had an mtu of 1438. Once again, I really have no idea how these are both getting determined and then set. If I change the home node's mtu to 1438 to match the exit server, all of a sudden there are no more issues getting ssh connections across the link.

I know that we can figure out what the hell tunneldigger is doing if we go over the code thoroughly enough. Considering it's such an integral part of our project at the moment, I think we may have to just bite that bullet... @papazoga - sound fun? :P

max-b commented 9 years ago

Ok the broker is getting an mtu set in the up_hook which looks like we've statically assigned it. That was dumb. I think I've fixed it with this in the exit node.... https://github.com/sudomesh/exitnode/commit/0d3ef9606f8b89afd5bbe15c3462b41434020e4a

max-b commented 9 years ago

This looks fixed, and I've added an issue to the exit-node repo to add a script for any mtu changed hook.