search

bmx-routing / bmx7

BMX7 / SEMTOR Securely Entrusted Mesh Routing Protocol
GNU General Public License v2.0
63 stars 15 forks source link

tunnel mtu problems #30

Open axn opened 6 years ago

axn commented 6 years ago

MTU discovery does NOT work with bmx7 tunnels on openwrt devices

axn commented 6 years ago

The problem is that BMX7 by default uses a faked outer-ip-in-ip6 tunnel source address which prevents ICMPv6 Packet Too Big messages to reach the intended origin of the MTU-exceeding packet.

The problem need fixes on two sides:

1) Openwrt currently ships a patched ip4-in-ip6 kernel module that adds support for MAP-E-FMRs mesh mode. This patch actually breaks the possibility of using an ip4ip6 tunnel interface as a fall back interface accepting ip4-in-ip6 tunneled packets from any remote address which works out of the box with any normal (non-patched) kernel and can be configured by setting up an 'ip -6 tunnel' with type 'any' or 'ip4ip6' and a remote address of '::'. I pushed a tentative fix for Re-Enabling any-remote tun6 addresses again for openwrt-18.06 here: https://github.com/axn/openwrt/commit/1b8b2644cf1ea59074ace31e61c5e588ca6bff86 and for openwrt-master here: https://github.com/axn/openwrt/commit/8ea3fc4f808fe5e9aef50d082bd19095a3113750

2) BMX7 must be modified to use its primary IPv6 address as outer ip-in-ip6 tunnel header. Again, a tentative commit can be found here: https://github.com/bmx-routing/bmx7/commit/d854135cf6795297221ae0de479eebd9a34b18f5 which can be enabled via --tunAnySrc=1

@aparcar can you check if this fixes the problem reported by you?

aparcar commented 6 years ago 
root@meshrc-e8b7b4:/etc/tinc/librenet6/hosts# bmx7 --tunAnySrc=1
[27585        8] ERROR apply_stream_opts: invalid argument: tunAnySrc=1

Am I missing something? It's freshly compiled

aparcar commented 6 years ago

I setup the following:

After all, I can't seem to start BMX7 accepting the tunAnySrc parameter. The 666 patch patch doesn't seem to break anything and your AllowZeroTunAddr patch is broken? Honestly I'm a bit confused.

axn commented 6 years ago

TunAnySrc parameter is only give with bmx7_tun.so plugin. So try bmx7 plugin=bmx7_tun.so tunAnySrc=1

Am 25. Mai 2018 19:55:17 MESZ schrieb Paul Spooren notifications@github.com:

root@meshrc-e8b7b4:/etc/tinc/librenet6/hosts# bmx7 --tunAnySrc=1
[27585        8] ERROR apply_stream_opts: invalid argument: tunAnySrc=1

Am I missing something? It's freshly compiled

-- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: https://github.com/bmx-routing/bmx7/issues/30#issuecomment-392134149

-- Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

axn commented 6 years ago

This works:

root@mlc1000:~# bmx7 -cpd8
PARAMETERS:
 plugin                 bmx7_config.so       (0) 
 plugin                 bmx7_json.so         (0) 
 plugin                 bmx7_sms.so          (0) 
 plugin                 bmx7_tun.so          (0) 
 plugin                 bmx7_topology.so     (0) 
 plugin                 bmx7_table.so        (0) 
 dev                    eth1                 (0) 
 unicastHna             2013:0:0:1000::/64   (0) 
 tunAnySrc              1                    (0) 
 tunDev                 default              (0) 
    /tun4Address        10.20.0.1/24         (0) 
    /tun6Address        2013:0:0:1000::1/64  (0) 
 tunOut                 ip6                  (0) 
    /network            2013::/16            (0) 
 tunOut                 ip4                  (0) 
    /network            10.20.0.0/16         (0) 
STATUS:
shortId  name    nodeKey cv revision primaryIp                               tun6Address         tun4Address  uptime     cpu txQ  nbs rts nodes 
F93ED1A4 mlc1000 RSA2048 21 d854135  fd70:f93e:d1a4:7ff8:dd9d:2404:1ec6:afa8 2013:0:0:1000::1/64 10.20.0.1/24 0:00:02:51 0.1 0/50 1   2   3/3   
INTERFACES:
dev  state linkKey    linkKeys          type     channel rateMax idx localIp                   rts helloSqn rxBpP   txBpP   
eth1 UP    DH2048M112 RSA896,DH2048M112 ethernet 0       1000M   1   fe80::a2cd:efff:fe10:1/64 2   26750    197/1.2 143/1.3 
LINKS:
shortId  name    linkKey    linkKeys          nbLocalIp                dev  rts rq  tq  txRate wTxRate wTxRateEff wTxThr wTxThrEff mcs sgi chw wSnr 
4307C006 mlc1001 DH2048M112 RSA896,DH2048M112 fe80::a2cd:efff:fe10:101 eth1 2   100 100 1000M  -1      -1         -1     -1        0   0   20  0   
ORIGINATORS:
shortId  name    as S s T t descSqn lastDesc descSize cv revision primaryIp                               dev  nbShortId nbName  metric hops ogmSqn lastRef 
F93ED1A4 mlc1000 nQ A A A A 612     171      705+768  21 d854135  fd70:f93e:d1a4:7ff8:dd9d:2404:1ec6:afa8 ---  ---       ---     257G   0    25     5       
4307C006 mlc1001 nA A A A A 412     170      705+768  21 d854135  fd70:4307:c006:1d87:72fd:f7d4:e2d9:48a8 eth1 4307C006  mlc1001 999M   1    26     0       
9BC3537B mlc1002 pA A A A A 513     22       705+768  21 d854135  fd70:9bc3:537b:6fb4:ca32:d887:9d9d:3047 eth1 4307C006  mlc1001 706M   2    4      4       
TUNNELS:
tunOut id       gwName net          min max hyst rating minBw tunName     tunRoute           remoteId remoteName advNet             advBw pathMtc tunMtc 
ip4    00000000 ---    10.20.0.0/16 16  128 20   100    960   X7C4default 10.20.1.0/24       4307C006 mlc1001    10.20.1.0/24       257G  999M    1199M  
ip4    00000000 ---    10.20.0.0/16 16  128 20   100    960   X7C4default 10.20.2.0/24       9BC3537B mlc1002    10.20.2.0/24       257G  706M    847M   
ip6    00000000 ---    2013::/16    16  128 20   100    960   X7C6default 2013:0:0:1001::/64 4307C006 mlc1001    2013:0:0:1001::/64 257G  999M    1199M  
ip6    00000000 ---    2013::/16    16  128 20   100    960   X7C6default 2013:0:0:1002::/64 9BC3537B mlc1002    2013:0:0:1002::/64 257G  706M    847M   
root@mlc1000:~# ping 10.20.1.1
PING 10.20.1.1 (10.20.1.1) 56(84) bytes of data.
64 bytes from 10.20.1.1: icmp_seq=1 ttl=63 time=2.43 ms
64 bytes from 10.20.1.1: icmp_seq=2 ttl=64 time=0.097 ms
^C
--- 10.20.1.1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.097/1.265/2.433/1.168 ms
root@mlc1000:~#
altergui commented 5 years ago

here testing the code with @g10h4ck, here is our result:

the "old" bmx7 (without tunAnySrc commit, or also with tunAnySrc commit included but without specifying tunAnySrc=1) produces these tunnel interfaces:

34: X7main@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1460 qdisc noqueue state UNKNOWN group default qlen 1
    link/tunnel6 fd70:4887:bc2d:ce6f:7dad:f423:8323:c593 peer fd71:4887:bc2d:ce6f:7dad:f423:8323:c593
    inet 10.82.172.170/32 scope global X7main
       valid_lft forever preferred_lft forever
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::d0bf:45ff:fedc:bcb5/64 scope link 
       valid_lft forever preferred_lft forever
[...snip...]
46: X7Out_570B504A@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1358 qdisc noqueue state UNKNOWN group default qlen 1
    link/tunnel6 fd71:570b:504a:601d:977f:dbd7:8c86:24b5 peer fd70:570b:504a:601d:977f:dbd7:8c86:24b5
    inet 10.82.172.170/32 scope global X7Out_570B504A
       valid_lft forever preferred_lft forever
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fd71:570b:504a:601d:977f:dbd7:8c86:24b5/128 scope global deprecated 
       valid_lft forever preferred_lft 0sec

X7main with source (real) fd70: and peer (fake) fd71: and X7Out with source (fake) fd71: and destination (real) fd70:

the "new" bmx7 (after applying tunAnySrc=1 like this in two nodes of different LibreMesh clouds):

root@LiMe-7aacaa:~# bmx7 -c plugin=bmx7_tun.so tunAnySrc=1
INFO  uci_save_option: bmx7.general.tunAnySrc=1
INFO  : --tunAnySrc               1                             
root@LiMe-7aacaa:~# killall bmx7 && sleep 10

(regarding the killall bmx7: if bmx7 is not fully restarted, the new X7Out interfaces will use real addresses but the X7main is not recreated and will still have a fake "peer")

successfully can ping to the other node over the tunnel!!!

root@LiMe-7aacaa:~# ip r
10.13.47.0/24 dev X7C4main proto static metric 1024 
10.82.172.0/24 dev br-lan proto kernel scope link src 10.82.172.170 
root@LiMe-7aacaa:~# ping -c 3 10.13.47.16
PING 10.13.47.16 (10.13.47.16) 56(84) bytes of data.
64 bytes from 10.13.47.16: icmp_req=1 ttl=64 time=8.47 ms
64 bytes from 10.13.47.16: icmp_req=2 ttl=64 time=2.14 ms
64 bytes from 10.13.47.16: icmp_req=3 ttl=64 time=2.14 ms

--- 10.13.47.16 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 2.142/4.254/8.476/2.985 ms
root@LiMe-7aacaa:~# ip r
10.13.47.0/24 dev X7Out_570B504A proto static metric 1024 
10.82.172.0/24 dev br-lan proto kernel scope link src 10.82.172.170

and this created tunnel X7Out is using real source addresses, and the X7main has an "empty" (::) peer, as can be seen

55: X7main@NONE: <NOARP,UP,LOWER_UP> mtu 1460 qdisc noqueue state UNKNOWN group default qlen 1
    link/tunnel6 fd70:4887:bc2d:ce6f:7dad:f423:8323:c593 brd ::
    inet 10.82.172.170/32 scope global X7main
       valid_lft forever preferred_lft forever
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::18e3:b8ff:fe83:8d0f/64 scope link 
       valid_lft forever preferred_lft forever
56: X7C4main: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet 10.82.172.170/32 scope global X7C4main
       valid_lft forever preferred_lft forever
    inet6 fe80::86a6:4b18:7028:40c3/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
57: X7C6main: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500
    link/none 
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::ae29:e69d:ea20:b0b2/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
58: X7Out_570B504A@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1358 qdisc noqueue state UNKNOWN group default qlen 1
    link/tunnel6 fd70:4887:bc2d:ce6f:7dad:f423:8323:c593 peer fd70:570b:504a:601d:977f:dbd7:8c86:24b5
    inet 10.82.172.170/32 scope global X7Out_570B504A
       valid_lft forever preferred_lft forever
    inet6 2012:0:0:aa::/128 scope global 
       valid_lft forever preferred_lft forever
    inet6 fd70:4887:bc2d:ce6f:7dad:f423:8323:c593/128 scope global deprecated 
       valid_lft forever preferred_lft 0sec
    inet6 fe80::3c07:cff:fe6d:43bb/64 scope link 
       valid_lft forever preferred_lft forever

so, we have just reproduced independently @axn test, and it looks great! thanks a lot

what's pending is to test this "in the real world" (or at least with a reduced MTU segment in the middle of the path) and see if it allows PMTUD to work correctly, finally ending the infamous "dead" tunnels

in particular, we could not compile bmx6 at this particular commit https://github.com/bmx-routing/bmx6/commit/5dc6678cf9c2887ca5e32c8d7527c5f660ddb7e9 on top of current openwrt, understandably since it's old code. But this means someone should port this tunAnySrc=1 patch to bmx6 as @pedro-nonfree suggested, so that we can compile a "fixed" bmx6 and test it in quintanalibre for example

axn commented 5 years ago

@altergui any updates from the "in the real world" tests of https://github.com/bmx-routing/bmx7/commit/d854135cf6795297221ae0de479eebd9a34b18f5 ? And did you discover any incompatibilities with current master. Otherwise I would like to merge https://github.com/bmx-routing/bmx7/commit/d854135cf6795297221ae0de479eebd9a34b18f5 into master and close this issue...

axn commented 5 years ago

I added https://github.com/bmx-routing/bmx7/commit/d854135cf6795297221ae0de479eebd9a34b18f5 plus another https://github.com/bmx-routing/bmx7/commit/cc245a22e54b5e110384485f18d9b805c91d8b71 patch to testing https://github.com/bmx-routing/bmx7/commits/testing branch.

The latter commit replaces tunAnySrc=<0|1> with tunRealSrc=<0|1|2> parameter which should be used as follows:

In the long term I'll make tunRealSrc=1 and later tunRealSrc=2 the default...

Hope this helps to smoothly get away with asymmetric tunnels and faked tunIps

Please @altergui use and report

dangowrt commented 5 years ago

Just updated the PR for openwrt-routing/packages.git after some testing: https://github.com/openwrt-routing/packages/pull/454