search

openwrt / openwrt

This repository is a mirror of https://git.openwrt.org/openwrt/openwrt.git It is for reference only and is not active for check-ins. We will continue to accept Pull Requests here. They will be merged via staging trees then into openwrt.git.
Other
20.25k stars 10.48k forks source link

6.6: bmips NAT64 performance degradated #15848

Closed GoetzGoerisch closed 4 months ago

GoetzGoerisch commented 4 months ago

Describe the bug

Using 5.15.158 give about 45Mbit/s throughput a NAT64 jool instance setup according NAT64 option 2.

Typical NAT44 throughput 90Mbit/s Load 1.5 Typical IPv6 throughput 100-120Mbit/s Load 1.5

Upgrading to SNAPSHOT 6.6.35 give now only about 10Mbit/s NAT64 througput, Load 0.3-0.4

Therefore the CPU is not loaded as much.

CC: @Noltari

OpenWrt version

r26815-b4069fad02

OpenWrt release

SNAPSHOT

OpenWrt target/subtarget

bmips/bcm6368

Device

Netgear DGND3800B

Image kind

Official downloaded image

Steps to reproduce

Actual behaviour

Throughput only about ~25% of 5.15.158

Expected behaviour

Throughput about the same

Additional info

Packet Steering or Software Offloading has no effect

root@OpenWrt:~# owut blob
{
  "client": "owut/2024.06.24~1b222bdc-r1",
  "target": "bmips/bcm6368",
  "profile": "netgear_dgnd3800b",
  "version": "SNAPSHOT",
  "version_code": "r26815-b4069fad02",
  "filesystem": "squashfs",
  "diff_packages": true,
  "packages": [
    "base-files",
    "busybox",
    "ca-bundle",
    "dnsmasq",
    "dropbear",
    "ethtool",
    "firewall4",
    "ip-full",
    "iwinfo",
    "jool-tools-netfilter",
    "kmod-b43",
    "kmod-gpio-button-hotplug",
    "kmod-leds-gpio",
    "kmod-usb-ledtrig-usbport",
    "kmod-usb-ohci",
    "kmod-usb2",
    "kmod-veth",
    "libustream-mbedtls",
    "logd",
    "luci",
    "luci-ssl",
    "mtd",
    "nano",
    "odhcp6c",
    "odhcpd-ipv6only",
    "owut",
    "ppp",
    "ppp-mod-pppoe",
    "procd",
    "procd-seccomp",
    "procd-ujail",
    "uci",
    "urandom-seed",
    "urngd",
    "wpad-basic-mbedtls"
  ]
}

Diffconfig

No response

Terms

GoetzGoerisch commented 4 months ago

So far I have had also 3 kernel panics.

The completes trace was this

Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.567316] skb len=204 headroom=102 headlen=131 tailroom=0
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.567316] mac=(102,14) net=(116,20) trans=136
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.567316] shinfo(txflags=0 nr_frags=0 gso(size=89 type=393216 segs=2))
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.567316] csum(0x0 ip_summed=0 complete_sw=0 valid=0 level=1)
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.567316] hash(0xddbd747c sw=1 l4=0) proto=0x0800 pkttype=0 iif=10
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.597367] dev name=openwrt feat=0x020061164fdd19e9
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.602432] skb linear:   00000000: 7e e8 b9 21 71 de 32 d4 7e dd 99 23 08 00 45 00
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.610377] skb linear:   00000010: 00 be ab 1b 00 00 3e 11 8e b6 c0 a8 a4 02 17 3c
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.618241] skb linear:   00000020: c6 76 a1 bb 01 bb 00 aa bb 2a ec 00 00 00 01 08
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.626118] skb linear:   00000030: 0c 88 ac 73 40 80 0e 18 00 40 48 07 5d 1b 3d 85
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.633986] skb linear:   00000040: 31 83 f8 99 1c c9 36 83 b6 09 8f 1c 50 e3 e7 f8
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.641891] skb linear:   00000050: e2 12 80 76 aa 0b e8 14 fc e8 df 16 2a 8c bb c4
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.649778] skb linear:   00000060: a7 56 33 5f 0e 2c 58 8b 04 d8 d5 15 79 6a 0b 50
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.657656] skb linear:   00000070: 88 87 49 59 a8 40 59 46 61 da e3 bb 51 8c 6a 7e
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.665524] skb linear:   00000080: 50 23 0f
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.669938] ------------[ cut here ]------------
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.674664] WARNING: CPU: 0 PID: 0 at net/core/dev.c:3313 skb_warn_bad_offload+0x94/0x138
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.683110] openwrt: caps=(0x020061164fdd19e9, 0x0000000000000000)
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.689468] Modules linked in: pppoe ppp_async nft_fib_inet nf_flow_table_inet b43(O) pppox ppp_generic nft_reject_ipv6 nft_reject_ipv4 nft_reject_inet nft_reject nft_redir nft_quota nft_numgen nft_nat nft_masq nft_log nft_limit nft_hash nft_flow_offload nft_fib_ipv6 nft_fib_ipv4 nft_fib nft_ct nft_chain_nat nf_tables mac80211(O) cfg80211(O) slhc nfnetlink nf_reject_ipv6 nf_reject_ipv4 nf_nat nf_log_syslog nf_flow_table nf_conntrack libcrc32c crc_ccitt cordic compat(O) ledtrig_usbport jool_siit(O) jool(O) nf_defrag_ipv6 nf_defrag_ipv4 jool_common(O) veth sha512_generic seqiv sha3_generic jitterentropy_rng drbg md5 hmac geniv rng cmac leds_gpio ohci_platform ohci_hcd fsl_mph_dr_of ehci_platform ehci_fsl ehci_hcd gpio_button_hotplug(O) usbcore nls_base usb_common ssb crc32c_generic
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.761412] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G           O       6.6.35 #0
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.768997] Stack : 72633300 00000004 00000000 00000000 8180fb5c 80b70000 80a30000 808b97ec
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.777631]         809ad6d0 80a333f7 00000000 00000000 00000000 10008f00 8180fb10 81839380
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.786259]         00000000 00000000 808b97ec 8180fa40 00000000 00000000 ffffefff ffffffea
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.794887]         00000156 8180fa4c 00000156 809b2398 10008f01 808b97ec 8180fc10 8059911c
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.803516]         00000009 808f7280 809aa680 00000003 00001008 00000000 00000000 80b70000
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.812136]         ...
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.814645] Call Trace:
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.817154] [<80016b44>] show_stack+0x28/0xf0
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.821640] [<807bc2e0>] dump_stack_lvl+0x5c/0x7c
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.826474] [<8003a74c>] __warn+0xb0/0xe4
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.830595] [<8003a900>] warn_slowpath_fmt+0x180/0x188
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.835881] [<8059911c>] skb_warn_bad_offload+0x94/0x138
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.841350] [<805d9b8c>] __skb_gso_segment+0x134/0x1b4
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.846634] [<80599d24>] validate_xmit_skb.isra.0+0x154/0x3f4
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.852548] [<8059a8b0>] __dev_queue_xmit+0x1c4/0xde4
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.857744] [<8063cc50>] ip_finish_output2+0x230/0x784
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.863070] [<82bcef8c>] sendpkt_send+0xac/0x1c0 [jool_common]
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.869281] [<82bcbe68>] route6+0x360/0x378 [jool_common]
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.880930] [<82bc6078>] hook_ipv6+0x78/0xd0 [jool_common]
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.886723] [<8062a1f0>] nf_hook_slow+0x6c/0x13c
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.891444] [<806ee598>] ipv6_rcv+0x6c/0x148
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.895833] [<8059d148>] __netif_receive_skb_one_core+0x44/0x54
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.901926] [<8059d344>] process_backlog+0xbc/0x1e4
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.906943] [<8059de90>] __napi_poll+0x3c/0x1c8
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.911602] [<8059e6a8>] net_rx_action+0x2d4/0x340
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.916532] [<8003f978>] handle_softirqs+0x168/0x320
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.921639] [<8003fe20>] irq_exit+0x84/0xb4
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.925939] [<8042a064>] plat_irq_dispatch+0x68/0x98
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.931046] [<80011c30>] handle_int+0x150/0x15c
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.935704] [<80011aa0>] __r4k_wait+0x20/0x40
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.940186] [<807e7120>] r4k_wait+0x30/0x3c
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.944487] [<807e7218>] default_idle_call+0x28/0x5c
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.949593] [<80081d08>] do_idle+0xb0/0x150
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.953894] [<8008204c>] cpu_startup_entry+0x3c/0x44
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.959001] [<807e73f0>] kernel_init+0x0/0x110
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.963570]
Mon Jul  1 07:52:38 2024 kern.warn kernel: [  506.965155] ---[ end trace 0000000000000000 ]---

Look similar to #15821

Noltari commented 4 months ago

@GoetzGoerisch this was the only thing that was changed from v5.15 to v6.1 (and therefore v6.6): https://github.com/openwrt/openwrt/pull/15849/commits/920aacb5cc22168a49eea35d65d7e812e9b945c5

I hope this fixes the issue... If it isn't fixed it's probably another different issue coming from the new kernel version...

rany2 commented 4 months ago

There's also https://github.com/openwrt/openwrt/commit/b5c53848c3792355a2dd4ff0e06bf4d230e1f5fb#commitcomment-143014902

Noltari commented 4 months ago

@GoetzGoerisch please test the fix and reopen this issue if it's not fixed.

GoetzGoerisch commented 4 months ago

@Noltari not fixed. r26830-a7f37b401b Throughput is still subpar and kernel is crashing.

How do I produce more debugging output?

Noltari commented 4 months ago

@Noltari not fixed. r26830-a7f37b401b Throughput is still subpar and kernel is crashing.

Then it's probably a general issue, as @rany2 pointed out...