conntrack-tools: kernel panic in nf_tables on OpenWrt 22.03.4+

[lowjoel]

Maintainer: @jow Environment: OpenWrt 22.03.4+, Linksys E8450 (aarch64)

Description: Hello there, I have conntrack -E running on my router to have some kind of firewall-like behaviour for a protocol that's not natively supported. Ever since upgrading to OpenWrt 22.03.4 (and also OpenWrt 22.03.5), I've been getting kernel panics from nf_tables:

On 22.03.4 (https://forum.openwrt.org/t/belkin-rt3200-linksys-e8450-wifi-ax-discussion/94302/3583?u=lowjoel)

<1>[ 2177.211289] Unable to handle kernel NULL pointer dereference at virtual address 000000000000001c
<1>[ 2177.220112] Mem abort info:
<1>[ 2177.222898]   ESR = 0x96000006
<1>[ 2177.225943]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[ 2177.231261]   SET = 0, FnV = 0
<1>[ 2177.234305]   EA = 0, S1PTW = 0
<1>[ 2177.237435] Data abort info:
<1>[ 2177.240315]   ISV = 0, ISS = 0x00000006
<1>[ 2177.244141]   CM = 0, WnR = 0
<1>[ 2177.247101] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000041721000
<1>[ 2177.253538] [000000000000001c] pgd=00000000401b2003, p4d=00000000401b2003, pud=00000000401b2003, pmd=0000000000000000
<0>[ 2177.264222] Internal error: Oops: 96000006 [#1] SMP
<7>[ 2177.269090] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount xt_state xt_helper xt_conntrack xt_connmark xt_connbytes xt_CT pppox ppp_generic nft_redir nft_nat nft_masq nft_flow_offload nft_fib_inet nft_ct nft_chain_nat nf_nat nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet nf_flow_table nf_conntrack_netlink nf_conntrack mt7915e mt7615e mt7615_common mt76_connac_lib mt76 mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_policy xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_esp xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY xfrm_interface slhc sch_cake nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_quota nft_objref nft_numgen nft_log nft_limit nft_hash nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_counter nf_tables nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 macvlan libcrc32c iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ECN ip_tables hwmon crc_ccitt
<7>[ 2177.269257]  compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 tunnel6 tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo sha1_generic seqiv md5 echainiv des_generic libdes cbc authenc leds_gpio xhci_plat_hcd gpio_button_hotplug
<7>[ 2177.426081] CPU: 0 PID: 16503 Comm: nft Tainted: G S                5.10.176 #0
<7>[ 2177.433379] Hardware name: Linksys E8450 (UBI) (DT)
<7>[ 2177.438249] pstate: 00000005 (nzcv daif -PAN -UAO -TCO BTYPE=--)
<7>[ 2177.444266] pc : 0xffffffc008a61a28 [nf_tables@0000000030e5fcf7+0x24000]
<7>[ 2177.450961] lr : 0xffffffc008a61a18 [nf_tables@0000000030e5fcf7+0x24000]
<7>[ 2177.457651] sp : ffffffc01324b5b0
<7>[ 2177.460957] x29: ffffffc01324b5b0 x28: ffffffc008a5393c
<7>[ 2177.466261] x27: 0000000000000000 x26: ffffffc008a53230
<7>[ 2177.471565] x25: ffffff800205d8d0 x24: 0000000000000002
<7>[ 2177.476870] x23: ffffff80025f8000 x22: ffffff8003ee2c00
<7>[ 2177.482174] x21: ffffff800205d800 x20: ffffff80036f5900
<7>[ 2177.487478] x19: 0000000000000000 x18: 0000000000000000
<7>[ 2177.492784] x17: 0000000000000000 x16: 0000000000000000
<7>[ 2177.498088] x15: 0000007f92b32690 x14: 0003000880020018
<7>[ 2177.503393] x13: 0411a8c000010008 x12: ffffffffffffffff
<7>[ 2177.508698] x11: 0000000000000040 x10: 0000000000000007
<7>[ 2177.514002] x9 : 0000000000000000 x8 : ffffff80025f9000
<7>[ 2177.519306] x7 : 0000000000000000 x6 : 000000000000003f
<7>[ 2177.524610] x5 : 0000000000000040 x4 : ffffffc01324b588
<7>[ 2177.529915] x3 : 0000000000000001 x2 : 0000000000000b20
<7>[ 2177.535220] x1 : 0000000000000000 x0 : 0000000000000018
<7>[ 2177.540524] Call trace:
<7>[ 2177.542966]  0xffffffc008a61a28 [nf_tables@0000000030e5fcf7+0x24000]
<7>[ 2177.549313]  0xffffffc008a5393c [nf_tables@0000000030e5fcf7+0x24000]
<7>[ 2177.555659]  0xffffffc008a53e24 [nf_tables@0000000030e5fcf7+0x24000]
<7>[ 2177.562016]  0xffffffc008900844 [nfnetlink@00000000cd5bbb49+0x3000]
<7>[ 2177.568277]  0xffffffc008900e3c [nfnetlink@00000000cd5bbb49+0x3000]
<7>[ 2177.574533]  0xffffffc0106c7f00
<7>[ 2177.577664]  0xffffffc0106c8190
<7>[ 2177.580795]  0xffffffc010637180
<7>[ 2177.583927]  0xffffffc01063a508
<7>[ 2177.587059]  0xffffffc01063a5c4
<7>[ 2177.590190]  0xffffffc01063a630
<7>[ 2177.593321]  0xffffffc010010ef0
<7>[ 2177.596453]  0xffffffc010010fbc
<7>[ 2177.599584]  0xffffffc010809174
<7>[ 2177.602715]  0xffffffc010809590
<7>[ 2177.605846]  0xffffffc0100025c8
<0>[ 2177.608982] Code: aa0003f7 b40017e0 b1006260 540003a0 (39401001)
<4>[ 2177.615066] ---[ end trace dbc3deb609cf28ad ]---

On 22.03.5

<1>[  320.078326] Unable to handle kernel NULL pointer dereference at virtual address 000000000000001c
<1>[  320.087136] Mem abort info:
<1>[  320.089923]   ESR = 0x96000006
<1>[  320.092969]   EC = 0x25: DABT (current EL), IL = 32 bits
<1>[  320.098287]   SET = 0, FnV = 0
<1>[  320.101333]   EA = 0, S1PTW = 0
<1>[  320.104474] Data abort info:
<1>[  320.107348]   ISV = 0, ISS = 0x00000006
<1>[  320.111175]   CM = 0, WnR = 0
<1>[  320.114155] user pgtable: 4k pages, 39-bit VAs, pgdp=0000000040ad1000
<1>[  320.120652] [000000000000001c] pgd=0000000040bd1003, p4d=0000000040bd1003, pud=0000000040bd1003, pmd=0000000000000000
<0>[  320.131279] Internal error: Oops: 96000006 [#1] SMP
<7>[  320.136200] Modules linked in: xt_connlimit pppoe ppp_async nf_conncount xt_state xt_helper xt_conntrack xt_connmark xt_connbytes xt_CT pppox ppp_generic nft_redir nft_nat nft_masq nft_flow_offload nft_fib_inet nft_ct nft_chain_nat nf_nat nf_flow_table_ipv6 nf_flow_table_ipv4 nf_flow_table_inet nf_flow_table nf_conntrack_netlink nf_conntrack mt7915e mt7615e mt7615_common mt76_connac_lib mt76 mac80211 ipt_REJECT cfg80211 xt_time xt_tcpudp xt_tcpmss xt_statistic xt_recent xt_policy xt_multiport xt_mark xt_mac xt_limit xt_length xt_hl xt_esp xt_ecn xt_dscp xt_comment xt_TCPMSS xt_LOG xt_HL xt_DSCP xt_CLASSIFY xfrm_interface slhc sch_cake nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_quota nft_objref nft_numgen nft_log nft_limit nft_hash nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_counter nf_tables nf_reject_ipv4 nf_log_ipv6 nf_log_ipv4 nf_log_common nf_defrag_ipv6 nf_defrag_ipv4 macvlan libcrc32c iptable_raw iptable_mangle iptable_filter ipt_ah ipt_ECN ip_tables hwmon crc_ccitt
<7>[  320.136368]  compat sch_tbf sch_ingress sch_htb sch_hfsc em_u32 cls_u32 cls_route cls_matchall cls_fw cls_flow cls_basic act_skbedit act_mirred act_gact xt_set ip_set_list_set ip_set_hash_netportnet ip_set_hash_netport ip_set_hash_netnet ip_set_hash_netiface ip_set_hash_net ip_set_hash_mac ip_set_hash_ipportnet ip_set_hash_ipportip ip_set_hash_ipport ip_set_hash_ipmark ip_set_hash_ipmac ip_set_hash_ip ip_set_bitmap_port ip_set_bitmap_ipmac ip_set_bitmap_ip ip_set nfnetlink ip6table_mangle ip6table_filter ip6_tables ip6t_REJECT x_tables nf_reject_ipv6 ifb sit ipcomp6 xfrm6_tunnel esp6 ah6 xfrm4_tunnel ipcomp esp4 ah4 tunnel6 tunnel4 ip_tunnel xfrm_user xfrm_ipcomp af_key xfrm_algo sha1_generic seqiv md5 echainiv des_generic libdes cbc authenc leds_gpio xhci_plat_hcd gpio_button_hotplug
<7>[  320.293202] CPU: 1 PID: 16460 Comm: nft Tainted: G S                5.10.176 #0
<7>[  320.300504] Hardware name: Linksys E8450 (UBI) (DT)
<7>[  320.305375] pstate: 00000005 (nzcv daif -PAN -UAO -TCO BTYPE=--)
<7>[  320.311396] pc : 0xffffffc008a4fa28 [nf_tables@00000000ac52311a+0x24000]
<7>[  320.318096] lr : 0xffffffc008a4fa18 [nf_tables@00000000ac52311a+0x24000]
<7>[  320.324788] sp : ffffffc014ed35b0
<7>[  320.328095] x29: ffffffc014ed35b0 x28: ffffffc008a4193c
<7>[  320.333400] x27: 0000000000000000 x26: ffffffc008a41230
<7>[  320.338705] x25: ffffff800346e6d0 x24: 0000000000000001
<7>[  320.344009] x23: ffffff8002855000 x22: ffffff8000850d00
<7>[  320.349314] x21: ffffff800346e600 x20: ffffff8006028b80
<7>[  320.354619] x19: 0000000000000000 x18: 0000000000000000
<7>[  320.359924] x17: 0000000000000000 x16: 0000000000000000
<7>[  320.365228] x15: 0000007fbac12670 x14: 0003000880020018
<7>[  320.370533] x13: d211a8c000010008 x12: ffffffffffffffff
<7>[  320.375838] x11: 0000000000000040 x10: 0000000000000007
<7>[  320.381143] x9 : 0000000000000000 x8 : ffffff8002856000
<7>[  320.386447] x7 : 0000000000000000 x6 : 000000000000003f
<7>[  320.391751] x5 : 0000000000000040 x4 : ffffffc014ed3588
<7>[  320.397056] x3 : 0000000000000001 x2 : 0000000000000b20
<7>[  320.402362] x1 : 0000000000000000 x0 : 0000000000000018
<7>[  320.407668] Call trace:
<7>[  320.410118]  0xffffffc008a4fa28 [nf_tables@00000000ac52311a+0x24000]
<7>[  320.416469]  0xffffffc008a4193c [nf_tables@00000000ac52311a+0x24000]
<7>[  320.422819]  0xffffffc008a41e24 [nf_tables@00000000ac52311a+0x24000]
<7>[  320.429183]  0xffffffc008900844 [nfnetlink@00000000070724b7+0x3000]
<7>[  320.435449]  0xffffffc008900e3c [nfnetlink@00000000070724b7+0x3000]
<7>[  320.441705]  0xffffffc0106c7ef0
<7>[  320.444838]  0xffffffc0106c8180
<7>[  320.447970]  0xffffffc010637170
<7>[  320.451101]  0xffffffc01063a4f8
<7>[  320.454233]  0xffffffc01063a5b4
<7>[  320.457364]  0xffffffc01063a620
<7>[  320.460495]  0xffffffc010010ef0
<7>[  320.463627]  0xffffffc010010fbc
<7>[  320.466758]  0xffffffc01080a164
<7>[  320.469889]  0xffffffc01080a580
<7>[  320.473020]  0xffffffc0100025c8
<0>[  320.476157] Code: aa0003f7 b40017e0 b1006260 540003a0 (39401001)
<4>[  320.482241] ---[ end trace a942e25d324195df ]---

Specifically, conntrack -E --proto udp --orig-dst 239.255.255.0 together with smcroute for that broadcast (to SSDP across vlan segments), coupled with an SSDP discovery packet seems to be triggering the crash.

Edit history:

I believe it's conntrack -E because once I stop the process (running as a service), my router no longer panics (has been up for about 5+hours; with conntrack -E running, I've had 3 panics in a span of about 30 mins). Any suggestions/ideas welcome.

Edit 1: After starting conntrack -E, it panicked within 3 minutes. I'm now trying with only one instance of conntrack -E to see if it makes a difference (they are all looking at different ports/IP address ranges) Edit 2: I tried using the debug kernel package on the device download page and tried to extract nf_tables.ko (which is significantly bigger). I can't insmod/modprobe it though? How do I get the right debug information?

[lowjoel]

Edit 3: I can run multiple copies of conntrack -E until I start filtering on 239.255.255.250 UDP (which is the SSDP address). Running that seems to trigger the panic.

[brada4]

Does it happen if you remove iptables modules and stay in nftables-only world?

[lowjoel]

Unfortunately I can't remove iptables because that router uses mwan3.

I should however add that the SSDP address had special routing added by smcroute to cross vlans. The crash can be triggered when an SSDP discovery packet is sent by any device.

Edit 4: conntrack -E with --orig-dst 239.255.255.0 without smcroute seems to not cause the kernel to crash.

[lowjoel]

cc @mwarning ☝️ I'm not sure if you're able to put two and two together in this instance. There's probably some funky interaction with the newer patches in 5.10. Not sure if 5.15 is affected.

[brada4]

mwan3 works with iptables-nft. please check installing iptables-nft before mwan3

[brada4]

One bug is almost-null pointer dereference. Other if you confirmed is that iptables dependency unnecesarily pulls iptables-legacy.

[lowjoel]

I will try to remove all the iptables modules. I got iptables-nft in, but I still see the old iptable modules installed, need to dig why. I'll do that then return with whether I can still get it to panic.

[brada4]

I cannot get it to crash - running 5 copies of conntrack -E and miniupnpd while windows client exploring proximity devices via multicast. Probably some detail is missing in general picture.

The legacy warning is not harmful as long as nothing is programmed in legacy iptables, e.g. their frontends not installed at all. I managed to get rid of legacy warning here: https://github.com/openwrt/openwrt/issues/10988 (x_tables is loaded by iptables-nft when first used alone it does not trigger warning later)

[lowjoel]

did you also run smcroute? my simplified smcroute.conf:

phyint br-lan.vlan1 enable ttl-threshold 2
phyint br-lan.vlan2 enable ttl-threshold 2

mgroup from br-lan.vlan1 group 239.255.255.250
mroute from br-lan.vlan1 group 239.255.255.250 to br-lan.vlan2

(basically I'm trying to get my devices in a vlan2 to also reply to the SSDP solicitation from vlan1)

It seems I needed smcroute, conntrack -E, and the SSDP packet to cause the panic. Two of the 3 will work.

[brada4]

I have set up pretend-repeater minus mwan3 and waiting for crash seems sometimes multiple identical multicast states appear though I dont have base view on how it should be.

I base my claim that mwan3 setup is unstable on Ubuntu/Debian asking reboot after switching between iptables-? worlds to keep from mixing modules, them referring unpredictable behaviour upstream without exact pointer.

mwan3 needs only xt_ipset besides ip6tables-nft and iptables-nft, certainly not ip_tables kmod that triggers legacy warning in -nft tools. Basically you need to

• make sure iptables and ip6tables are -nft- versions
• remove iptables and xt????? modules via rmmod, checking dependencies in lsmod (leaving xt_ipset used by mwan3 intact)
• try crashing again

[lowjoel]

iptables/ip6tables are nft versions:

#  iptables -V
iptables v1.8.7 (nf_tables)
# ip6tables -V
ip6tables v1.8.7 (nf_tables)

I tried to remove the ip_tables and ip6_tables modules:

# for i in iptable_raw iptable_mangle iptable_filter ip_tables ip6table_mangle ip6table_filter ip6_tables; do rmmod $i; done
unloading the module failed
# lsmod | grep -E 'ip_|ip6_'
ip_set                 32768 17 xt_set,ip_set_list_set,ip_set_hash_netportnet,ip_set_hash_netport,ip_set_hash_netnet,ip_set_hash_netiface,ip_set_hash_net,ip_set_hash_mac,ip_set_hash_ipportnet,ip_set_hash_ipportip,ip_set_hash_ipport,ip_set_hash_ipmark,ip_set_hash_ipmac,ip_set_hash_ip,ip_set_bitmap_port,ip_set_bitmap_ipmac,ip_set_bitmap_ip
ip_set_bitmap_ip       16384  0
ip_set_bitmap_ipmac    16384  0
ip_set_bitmap_port     16384  0
ip_set_hash_ip         32768  0
ip_set_hash_ipmac      32768  0
ip_set_hash_ipmark     32768  0
ip_set_hash_ipport     32768  0
ip_set_hash_ipportip   32768  0
ip_set_hash_ipportnet   40960  0
ip_set_hash_mac        20480  0
ip_set_hash_net        36864  5
ip_set_hash_netiface   36864  0
ip_set_hash_netnet     40960  0
ip_set_hash_netport    36864  0
ip_set_hash_netportnet   40960  0
ip_set_list_set        16384  1
ip_tunnel              20480  1 sit
ip6_tables             28672  5
nfnetlink              12288  9 nf_conntrack_netlink,nft_compat,nf_tables,ip_set
x_tables               24576 35 xt_connlimit,xt_state,xt_helper,xt_conntrack,xt_connmark,xt_connbytes,xt_CT,ipt_REJECT,xt_time,xt_tcpudp,xt_tcpmss,xt_statistic,xt_recent,xt_policy,xt_multiport,xt_mark,xt_mac,xt_limit,xt_length,xt_hl,xt_esp,xt_ecn,xt_dscp,xt_comment,xt_TCPMSS,xt_LOG,xt_HL,xt_DSCP,xt_CLASSIFY,nft_compat,ipt_ah,ipt_ECN,xt_set,ip6_tables,ip6t_REJECT

So x_tables are still using ip6tables. Is that OK? I don't know which xt??? modules to remove though, they seem to be part of the normal nftables flow?

# lsmod | grep 'xt_'
xt_CLASSIFY            12288  0
xt_CT                  12288  0
xt_DSCP                12288  0
xt_HL                  12288  0
xt_LOG                 12288  0
xt_TCPMSS              12288  0
xt_comment             12288 18
xt_connbytes           12288  0
xt_connlimit           12288  0
xt_connmark            12288  4
xt_conntrack           12288  0
xt_dscp                12288  0
xt_ecn                 12288  0
xt_esp                 12288  0
xt_helper              12288  0
xt_hl                  12288  0
xt_length              12288  0
xt_limit               12288  0
xt_mac                 12288  0
xt_mark                12288 65
xt_multiport           12288  0
xt_policy              12288  0
xt_recent              20480  0
xt_set                 16384 15
xt_state               12288  0
xt_statistic           12288  0
xt_tcpmss              12288  0
xt_tcpudp              12288  0
xt_time                12288  0

I think I found something else, it's actually the nft command that causes the panic (and matches the stack trace):

# nft add element inet fw4 SSDP-Response { 239.255.255.250 }

Does that panic for you while running everything else?

Commands I have running (I'm going to edit while I try different combinations):

# conntrack -E --buffer-size 1048576 --proto tcp --orig-src 239.255.255.0 --orig-port-dst 1900 &
# while true; do \
nft delete element inet fw4 SSDP-Response { 239.255.255.250 } ; \
sleep 1; \
nft add element inet fw4 SSDP-Response { 239.255.255.250 } ; \
sleep 1; \
done

EDIT 5: It seems like smcroute and conntrack don't have to be running. Once there's an existing SSDP session, the kernel panics when the ipset has the SSDP address in a set. I wonder if I need to have firewall rules referencing that ipset?

/etc/config/firewall:

config ipset
        option name             SSDP-Response
        option match            dest_net
        option timeout          60
config rule
        option target           ACCEPT
        option src              vlan2
        option src_ip           xxx.yyy.zzz.aaa
        option dest             vlan1
        option ipset            SSDP-Response
        option proto            udp

[brada4]

Should not crash though.... Does not for me...

Textual interpretation of ruleset: When multicast request is sent to (other) network that network is added to timed ipset which is used to permit unicast responses back.

Just thinking aloud: I see in ruleset that ipset is with timeout? Maybe something to do with removal on timeout and (absent) locking? Just thinking aloud. The problem is to get debug version next to the issue - VM has space for debuginfo but too much speed to hit (?locking) issue.

I think ton of xt_ modules is just cosmetic defect. Only one around ipsets does something here.

[lowjoel]

Possibly, the best would be for me to get a debug version of nf_tables.ko but I can't figure out how to build one and replace it on the router.

Before that, I can try running this without the timeout and see if I can replicate. I did a diff between kernels 5.10.161 and 5.10.176 and the only major change is a bugfix in the rbtree implementation. But I don't have enough context to know how this is related (or not)

[brada4]

That would indeed isolate problem to changing ipset or not.

[lowjoel]

So disabling the timeout seems to give me much better uptime. It's been adding and removing elements to the SSDP-Response set for about 24h now without problems. With a timeout, it'll panic in less than an hour. So we're definitely onto something.

[brada4]

In your repeatable setup having expiring ipset triggers the elusive problem.

I thinking how to stress expiring ipset with updates so that ipset update or access or scheduled expiration meet as often as possible to have generic kernel problem repeater.

[lowjoel]

I just compiled nf_tables.ko from source, and it seems to be different from the one that is provided in the debug tarball. I can't test it right now, but I'll update here if I can and get a nice stack trace out of it.

[lowjoel]

I got a build of nf_tables.ko with debug info but it seems that the normal boot image that is distributed on every release doesn't seem to have CONFIG_KALLSYMS enabled, which means that the debug info isn't being used when printing the stack trace. I'm trying out whether I can boot a custom kernel with CONFIG_KALLSYMS (using kexec) just to trigger the crash, then go back the normal kernel (as a failsafe)

[brada4]

You dont have to debug in place, somebody better versed in nftables internals can decode function name from (stripped) file offsets and the code pattern in that place in file. You have found a workaround, if you get bored later you may flip the timeout with later version and see if crashes repeat.