MTU issues with Telekom Hybrid / Speedport Pro Plus when using mesh-on-wan

[T-X]

Bug report

What is the problem?

A Speedport Pro Plus at a Telekom Hybrid internet connection sends ICMPv6 Router Advertisements with an MTU option of 1440 bytes. This in turn breaks mesh-on-wan, causing packetloss for specific (but not the maximum, bc. of batman-adv fragmentation) byte ranges for client devices.

root@nml-wr1043nd-2:~# ip link show dev br-wan
7: br-wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether b0:48:7a:e7:f3:1e brd ff:ff:ff:ff:ff:ff
root@nml-wr1043nd-2:~# cat /proc/sys/net/ipv6/conf/br-wan/mtu 
1440

tcpdump capture on br-wan for ICMPv6 RA:

00:05:32.191685 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 64) fe80::1 > ff02::1: [icmp6 sum ok] ICMP6, router advertisement, length 64
    hop limit 255, Flags [other stateful], pref medium, router lifetime 180s, reachable time 30000ms, retrans timer 0ms
      prefix info option (3), length 32 (4): 2003:e2:e706:ec2a::/64, Flags [onlink, auto], valid time 172800s, pref. time 86400s
      mtu option (5), length 8 (1):  1440
      source link-address option (1), length 8 (1): 50:6f:0c:c7:80:a0

• What is not working as expected? How is it misbehaving?
  • Some internet pages don't load properly
  • Trying to do a node1$ ping6 -M do -s 1300 <node2-ipv6-addr> on the br-client layer is stuck, even though node1 and node2 are connected via mesh-on-wan; <1300 works as expected
  • Trying to do a node1$ ping6 -M do -s 1382 fe80::[...]%vx_mesh_wan to node2 returns: ping: local error: Message too long, mtu=1370
  • So interestingly here the Linux kernel seems to propagate the MTU issue to the IP stack, but not to the interface MTUs, so nothing usable for batman-adv. vx_mesh_wan interface MTUs look as follows:

    root@nml-wr1043nd-2:~# ip link show dev vx_mesh_wan
    13: vx_mesh_wan: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1430 qdisc noqueue master bat0 state UNKNOWN qlen 1000
    link/ether 66:c6:34:9d:58:40 brd ff:ff:ff:ff:ff:ff
    root@nml-wr1043nd-2:~# cat /proc/sys/net/ipv6/conf/vx_mesh_wan/mtu 
    1430

• When did the problem first start showing up?
  • The problem likely first started happening when we switched from TAL DSL to Telekom Hybrid
• What were you doing when you first noticed the problem?
  • Surfing web pages initially; occasional, temporary issues, depending on if mesh-on-wan links were chosen or not; then 100% reproduceable when adding a network cable and mesh-on-wan to the wifi router I'm connected to with my laptop
• On which devices (vendor, model and revision) is it misbehaving?
  • T-Com Speedport Pro Plus
• Does the issue appear on multiple devices or targets?
  • The issue appears for all clients connected to node which uses mesh-on-wan, where the WAN is shared with this Speedport router

What is the expected behaviour?

• How do you think it should work instead?
  • Instead of setting a fixed MTU of 1430 on vx_mesh_wan, both the overall interface MTU and the IPv6 specific interface MTU (/proc/sys/net/ipv6/conf/br-wan/mtu) should be taken into account at least. The issue is that the Speedport Pro Plus sends out an ICMPv6 RA with an MTU option of 1440 bytes. But does not set the MTU option in DHCPv4. Leading to a general interface MTU on br-wan of 1500, but /proc/sys/net/ipv6/conf/br-wan/mtu shows 1440. Which then leads to packetloss with batman-adv over the vx_mesh_wan interface for large packets, as VXLAN uses IPv6 for transport in Gluon. Probably, ideally the Linux kernel VXLAN implementation would take the lowered /proc/sys/net/ipv6/conf/br-wan/mtu into account, too, when using an IPv6 transport?
• Did it work like that before?
  • No

Gluon Version:

• v2021.1.2

Site Configuration:

https://git.chaotikum.org/freifunk-luebeck/site-ffhl/-/tree/v0.15.2

Edit: Ideally, the Speedport would not send this option at all. I don't like either the ICMPv6 RA MTU option or the DHCP MTU option. They are only causing trouble. If the router were only sending ICMPv6 "Packet Too Big" and ICMP "Destination unreachable - The datagram is too big." should be more reliable, sufficient and would precisely only affect internet routes. But the configuration options on the Speedport Pro Plus itself are very limited, there is not even an option to disable IPv6 or ICMPv6-RA for instance. So that's why maybe it could be useful to find a solution in Gluon or the Linux kernel for this.

[mweinelt]

Isn't the bug here that br-wan does not accept the advertise MTU? And if all nodes were on the same WAN they'd all use the same lower MTU, which should work, no?

[T-X]

@mweinelt br-wan does accept the MTU advertised by ICMPv6 RA - just not on the interface directly, which would be for all protocol families. But instead only on the IPv6 specific / for-IPv6-only interface MTU on /proc/sys/net/ipv6/conf/br-wan/mtu.

All nodes on the wan side get the same IPv6 MTU on br-wan. But the issue is that the vxlan interface still has an 1430 bytes MTU, which batman-adv will try to use, but which will not work.

[rubo77]

Will this problem also occur, if we have set an MTU of 1300 in our firmware site.conf?