search

pcengines / apu2-documentation

Documentation and scripts for building and adjusting PC Engines APU2 firmware
https://pcengines.github.io/apu2-documentation/
208 stars 46 forks source link

Kernel panic with ath10k #283

Open dseomn opened 2 years ago

dseomn commented 2 years ago

PCEngines firmware version v4.14.0.5

APU variant apu2e4

OS and OS Version Debian GNU/Linux 10 (buster)

Affected component(s), peripheral(s) or functionality The entire system crashes.

Brief summary The kernel panics occasionally. The watchdog still works though, so it gets rebooted and then works.

How reproducible I think it happens a bit less than once per week.

How to reproduce I haven't noticed any pattern of when it happens.

Expected behavior The kernel shouldn't panic.

Actual behavior The kernel panics. Here's a log from the serial console the most recent time it happened:

[86737.652272] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-110)
[259531.353847] wlp5s0: failed to remove key (0, 14:c1:4e:b7:b5:d2) from hardware (-110)
[432016.150262] wlp1s0.guest.h2: failed to remove key (2, ff:ff:ff:ff:ff:ff) from hardware (-11)
[518728.502333] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-110)
[1005496.119589] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-11)
[1295278.270050] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-11)
[1296309.161775] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-110)
[1471067.906042] wlp1s0.lan.l3: failed to remove key (0, a0:af:bd:e9:cc:c6) from hardware (-11)
[1555503.955474] wlp5s0: failed to remove key (0, 14:c1:4e:b7:b5:d2) from hardware (-110)
[1589501.336457] wlp1s0: failed to remove key (0, 14:c1:4e:b7:b5:d2) from hardware (-11)
[1589502.759769] BUG: unable to handle kernel NULL pointer dereference at 0000000000000fc0
[1589502.767792] PGD 0 P4D 0 
[1589502.770522] Oops: 0002 [#1] SMP NOPTI
[1589502.774376] CPU: 3 PID: 19198 Comm: kworker/3:0 Tainted: G        W   E     4.19.0-18-amd64 #1 Debian 4.19.208-1
[1589502.784721] Hardware name: PC Engines apu2/apu2, BIOS v4.14.0.5 10/11/2021
[1589502.791874] Workqueue: events_freezable ieee80211_restart_work [mac80211]
[1589502.798856] RIP: 0010:_raw_spin_lock_bh+0x17/0x30
[1589502.803741] Code: 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 81 05 10 e2 cd 4b 00 02 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 89 c6 e8 d9 33 99 ff 66 90 c3 66 0f 1f
[1589502.822666] RSP: 0018:ffff8eca2ab83cc0 EFLAGS: 00010246
[1589502.828074] RAX: 0000000000000000 RBX: 0000000000000fc0 RCX: 0000000000000000
[1589502.835389] RDX: 0000000000000001 RSI: ffff8eca27e1f4c8 RDI: 0000000000000fc0
[1589502.842702] RBP: ffff8eca27e1f3a8 R08: 0000000000000000 R09: 0000000000000000
[1589502.850019] R10: ffff8ec935c7e0f8 R11: 00000000000001ff R12: ffffffffc0b66030
[1589502.857332] R13: 0000000000000000 R14: ffff8eca26dc0ff8 R15: ffff8eca27e1e8c0
[1589502.864650] FS:  0000000000000000(0000) GS:ffff8eca2ab80000(0000) knlGS:0000000000000000
[1589502.872915] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1589502.878841] CR2: 0000000000000fc0 CR3: 00000001291ec000 CR4: 00000000000406e0
[1589502.886148] Call Trace:
[1589502.888780]  <IRQ>
[1589502.891026]  ath10k_wmi_tx_beacons_iter+0x24/0x140 [ath10k_core]
[1589502.897253]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589502.903622]  __iterate_interfaces+0x72/0x110 [mac80211]
[1589502.909136]  ieee80211_iterate_active_interfaces_atomic+0xd/0x10 [mac80211]
[1589502.916313]  ath10k_wmi_op_ep_tx_credits+0x20/0x40 [ath10k_core]
[1589502.922533]  ath10k_htc_process_trailer+0x1af/0x280 [ath10k_core]
[1589502.928853]  ath10k_htc_rx_completion_handler+0x97/0x130 [ath10k_core]
[1589502.935592]  ? ath10k_htc_process_trailer+0x280/0x280 [ath10k_core]
[1589502.942051]  ath10k_pci_process_rx_cb+0x174/0x1b0 [ath10k_pci]
[1589502.948112]  ath10k_ce_per_engine_service+0x93/0x100 [ath10k_core]
[1589502.954508]  ath10k_ce_per_engine_service_any+0x75/0xb0 [ath10k_core]
[1589502.961136]  ath10k_pci_napi_poll+0x44/0x100 [ath10k_pci]
[1589502.966724]  net_rx_action+0x149/0x3b0
[1589502.970666]  __do_softirq+0xde/0x2d8
[1589502.974439]  irq_exit+0xba/0xc0
[1589502.977772]  do_IRQ+0x7f/0xe0
[1589502.980933]  common_interrupt+0xf/0xf
[1589502.984778]  </IRQ>
[1589502.987094] RIP: 0010:ath10k_add_interface+0x79/0x1180 [ath10k_core]
[1589502.993628] Code: 89 e1 31 c0 48 c7 83 48 01 00 00 00 00 00 00 48 83 e7 f8 48 c7 83 a8 03 00 00 00 00 00 00 48 29 f9 81 c1 68 02 00 00 c1 e9 03 <f3> 48 ab 48 8b 83 30 01 00 00 48 85 c0 74 0c 48 8d 50 18 48 89 50
[1589503.012553] RSP: 0018:ffffb16bc0a5fd60 EFLAGS: 00000212 ORIG_RAX: ffffffffffffffda
[1589503.020301] RAX: 0000000000000000 RBX: ffff8eca27e1f3a8 RCX: 000000000000001f
[1589503.027616] RDX: ffff8eca237c0000 RSI: ffff8eca27e1f3a8 RDI: ffff8eca27e1f660
[1589503.034930] RBP: ffff8eca26dc0780 R08: 0000000000200000 R09: 0000000000000000
[1589503.042245] R10: 000000000000001f R11: ffff8eca2605a100 R12: ffff8eca27e1f4f0
[1589503.049558] R13: ffff8eca26dc2540 R14: ffff8eca24609f00 R15: ffff8eca26dc15a0
[1589503.056950]  drv_add_interface+0x4c/0x130 [mac80211]
[1589503.062208]  ieee80211_reconfig+0x226/0x1420 [mac80211]
[1589503.067622]  ? rcu_exp_wait_wake+0x240/0x240
[1589503.072080]  ? try_to_del_timer_sync+0x4d/0x80
[1589503.076780]  ieee80211_restart_work+0xbb/0xe0 [mac80211]
[1589503.082286]  process_one_work+0x1a7/0x3a0
[1589503.086484]  worker_thread+0x30/0x390
[1589503.090333]  ? create_worker+0x1a0/0x1a0
[1589503.094439]  kthread+0x112/0x130
[1589503.097856]  ? kthread_bind+0x30/0x30
[1589503.101702]  ret_from_fork+0x22/0x40
[1589503.105463] Modules linked in: cdc_ether usbnet r8152 mii cmac ctr ccm arc4 8021q garp mrp bridge stp llc wireguard(E) ip6_udp_tunnel udp_tunnel sr_mod cdrom nft_nat amd64_edac_mod ath10k_pci edac_mce_amd kvm_amd ath10k_core kvm evdev nft_fib_inet ath nft_fib_ipv4 nft_fib_ipv6 mac80211 irqbypass nft_fib uas crct10dif_pclmul crc32_pclmul leds_apu cfg80211 usb_storage ghash_clmulni_intel pcspkr sp5100_tco k10temp ccp sg fam15h_power rfkill rng_core nft_masq_ipv4 nft_masq button pcc_cpufreq nft_chain_nat_ipv4 nf_nat_ipv4 acpi_cpufreq nf_nat nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables_set nf_tables nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb sd_mod crc32c_intel ahci libahci sdhci_pci libata cqhci sdhci xhci_pci xhci_hcd mmc_core ehci_pci
[1589503.176812]  ehci_hcd scsi_mod aesni_intel igb usbcore aes_x86_64 crypto_simd cryptd glue_helper i2c_piix4 i2c_algo_bit usb_common dca thermal
[1589503.189799] CR2: 0000000000000fc0
[1589503.193307] ---[ end trace 90a3696deb151c81 ]---
[1589503.198112] RIP: 0010:_raw_spin_lock_bh+0x17/0x30
[1589503.202998] Code: 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 81 05 10 e2 cd 4b 00 02 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 89 c6 e8 d9 33 99 ff 66 90 c3 66 0f 1f
[1589503.221920] RSP: 0018:ffff8eca2ab83cc0 EFLAGS: 00010246
[1589503.227330] RAX: 0000000000000000 RBX: 0000000000000fc0 RCX: 0000000000000000
[1589503.234644] RDX: 0000000000000001 RSI: ffff8eca27e1f4c8 RDI: 0000000000000fc0
[1589503.241959] RBP: ffff8eca27e1f3a8 R08: 0000000000000000 R09: 0000000000000000
[1589503.249273] R10: ffff8ec935c7e0f8 R11: 00000000000001ff R12: ffffffffc0b66030
[1589503.256589] R13: 0000000000000000 R14: ffff8eca26dc0ff8 R15: ffff8eca27e1e8c0
[1589503.263905] FS:  0000000000000000(0000) GS:ffff8eca2ab80000(0000) knlGS:0000000000000000
[1589503.272170] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1589503.278100] CR2: 0000000000000fc0 CR3: 00000001291ec000 CR4: 00000000000406e0
[1589503.285415] Kernel panic - not syncing: Fatal exception in interrupt
[1589503.292021] Kernel Offset: 0x32c00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[1589503.302986] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
[1589503.310754] ------------[ cut here ]------------
[1589503.315555] sched: Unexpected reschedule of offline CPU#0!
[1589503.321235] WARNING: CPU: 3 PID: 19198 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x34/0x40
[1589503.330884] Modules linked in: cdc_ether usbnet r8152 mii cmac ctr ccm arc4 8021q garp mrp bridge stp llc wireguard(E) ip6_udp_tunnel udp_tunnel sr_mod cdrom nft_nat amd64_edac_mod ath10k_pci edac_mce_amd kvm_amd ath10k_core kvm evdev nft_fib_inet ath nft_fib_ipv4 nft_fib_ipv6 mac80211 irqbypass nft_fib uas crct10dif_pclmul crc32_pclmul leds_apu cfg80211 usb_storage ghash_clmulni_intel pcspkr sp5100_tco k10temp ccp sg fam15h_power rfkill rng_core nft_masq_ipv4 nft_masq button pcc_cpufreq nft_chain_nat_ipv4 nf_nat_ipv4 acpi_cpufreq nf_nat nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables_set nf_tables nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb sd_mod crc32c_intel ahci libahci sdhci_pci libata cqhci sdhci xhci_pci xhci_hcd mmc_core ehci_pci
[1589503.402194]  ehci_hcd scsi_mod aesni_intel igb usbcore aes_x86_64 crypto_simd cryptd glue_helper i2c_piix4 i2c_algo_bit usb_common dca thermal
[1589503.415161] CPU: 3 PID: 19198 Comm: kworker/3:0 Tainted: G      D W   E     4.19.0-18-amd64 #1 Debian 4.19.208-1
[1589503.425500] Hardware name: PC Engines apu2/apu2, BIOS v4.14.0.5 10/11/2021
[1589503.432636] Workqueue: events_freezable ieee80211_restart_work [mac80211]
[1589503.439608] RIP: 0010:native_smp_send_reschedule+0x34/0x40
[1589503.445273] Code: 05 61 58 0e 01 73 15 48 8b 05 b8 93 ea 00 be fd 00 00 00 48 8b 40 30 e9 ea a7 9b 00 89 fe 48 c7 c7 80 d9 a4 b4 e8 95 c3 6b 00 <0f> 0b c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 05 ed a3 5a
[1589503.464198] RSP: 0018:ffff8eca2ab83730 EFLAGS: 00010086
[1589503.469608] RAX: 0000000000000000 RBX: ffff8eca2aa22140 RCX: 0000000000000006
[1589503.476922] RDX: 0000000000000007 RSI: 0000000000000086 RDI: ffff8eca2ab966b0
[1589503.484236] RBP: ffff8eca24745a00 R08: 0000000000000518 R09: 000000000000000f
[1589503.491551] R10: 0000000000000000 R11: ffffffffb51fa6ef R12: ffff8eca2474615c
[1589503.498866] R13: ffff8eca2ab83780 R14: 0000000000000046 R15: 0000000000000000
[1589503.506181] FS:  0000000000000000(0000) GS:ffff8eca2ab80000(0000) knlGS:0000000000000000
[1589503.514447] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1589503.520375] CR2: 0000000000000fc0 CR3: 00000001291ec000 CR4: 00000000000406e0
[1589503.527682] Call Trace:
[1589503.530321]  <IRQ>
[1589503.532523]  check_preempt_curr+0x7a/0x90
[1589503.536720]  ttwu_do_wakeup+0x19/0x140
[1589503.540652]  try_to_wake_up+0x1d8/0x470
[1589503.544680]  autoremove_wake_function+0x11/0x50
[1589503.549398]  __wake_up_common+0x7a/0x190
[1589503.553506]  __wake_up_common_lock+0x7c/0xc0
[1589503.557962]  ? tick_sched_do_timer+0x60/0x60
[1589503.562414]  irq_work_run_list+0x4d/0x70
[1589503.566526]  update_process_times+0x4a/0x60
[1589503.570908]  tick_sched_handle+0x22/0x60
[1589503.575013]  tick_sched_timer+0x37/0x70
[1589503.579037]  __hrtimer_run_queues+0x100/0x280
[1589503.583580]  hrtimer_interrupt+0x100/0x210
[1589503.587859]  ? up+0x12/0x50
[1589503.590843]  smp_apic_timer_interrupt+0x6a/0x140
[1589503.595640]  apic_timer_interrupt+0xf/0x20
[1589503.599926] RIP: 0010:panic+0x204/0x24a
[1589503.603951] Code: eb a6 83 3d 9d cf ee 00 00 74 05 e8 46 8e 99 ff 48 c7 c6 00 1e 1f b5 48 c7 c7 20 74 a5 b4 e8 91 56 00 00 fb 66 0f 1f 44 00 00 <31> db e8 05 73 a3 ff 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 45 cf ee
[1589503.622876] RSP: 0018:ffff8eca2ab83ab0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[1589503.630624] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[1589503.637931] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff8eca2ab966b0
[1589503.645244] RBP: ffff8eca2ab83b20 R08: 0000000000000516 R09: 000000000000000f
[1589503.652550] R10: 0000000000000000 R11: ffffffffb51fa6ef R12: 0000000000000000
[1589503.659856] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000001
[1589503.667166]  ? apic_timer_interrupt+0xa/0x20
[1589503.671632]  oops_end.cold.9+0xc/0x18
[1589503.675481]  no_context+0x1be/0x380
[1589503.679156]  __do_page_fault+0xb2/0x4f0
[1589503.683179]  page_fault+0x1e/0x30
[1589503.686687] RIP: 0010:_raw_spin_lock_bh+0x17/0x30
[1589503.691570] Code: 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 81 05 10 e2 cd 4b 00 02 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 89 c6 e8 d9 33 99 ff 66 90 c3 66 0f 1f
[1589503.710495] RSP: 0018:ffff8eca2ab83cc0 EFLAGS: 00010246
[1589503.715903] RAX: 0000000000000000 RBX: 0000000000000fc0 RCX: 0000000000000000
[1589503.723210] RDX: 0000000000000001 RSI: ffff8eca27e1f4c8 RDI: 0000000000000fc0
[1589503.730526] RBP: ffff8eca27e1f3a8 R08: 0000000000000000 R09: 0000000000000000
[1589503.737840] R10: ffff8ec935c7e0f8 R11: 00000000000001ff R12: ffffffffc0b66030
[1589503.745154] R13: 0000000000000000 R14: ffff8eca26dc0ff8 R15: ffff8eca27e1e8c0
[1589503.752512]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589503.758830]  ath10k_wmi_tx_beacons_iter+0x24/0x140 [ath10k_core]
[1589503.765059]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589503.771421]  __iterate_interfaces+0x72/0x110 [mac80211]
[1589503.776920]  ieee80211_iterate_active_interfaces_atomic+0xd/0x10 [mac80211]
[1589503.784098]  ath10k_wmi_op_ep_tx_credits+0x20/0x40 [ath10k_core]
[1589503.790319]  ath10k_htc_process_trailer+0x1af/0x280 [ath10k_core]
[1589503.796628]  ath10k_htc_rx_completion_handler+0x97/0x130 [ath10k_core]
[1589503.803371]  ? ath10k_htc_process_trailer+0x280/0x280 [ath10k_core]
[1589503.809827]  ath10k_pci_process_rx_cb+0x174/0x1b0 [ath10k_pci]
[1589503.815880]  ath10k_ce_per_engine_service+0x93/0x100 [ath10k_core]
[1589503.822275]  ath10k_ce_per_engine_service_any+0x75/0xb0 [ath10k_core]
[1589503.828902]  ath10k_pci_napi_poll+0x44/0x100 [ath10k_pci]
[1589503.834491]  net_rx_action+0x149/0x3b0
[1589503.838434]  __do_softirq+0xde/0x2d8
[1589503.842203]  irq_exit+0xba/0xc0
[1589503.845536]  do_IRQ+0x7f/0xe0
[1589503.848692]  common_interrupt+0xf/0xf
[1589503.852543]  </IRQ>
[1589503.854853] RIP: 0010:ath10k_add_interface+0x79/0x1180 [ath10k_core]
[1589503.861386] Code: 89 e1 31 c0 48 c7 83 48 01 00 00 00 00 00 00 48 83 e7 f8 48 c7 83 a8 03 00 00 00 00 00 00 48 29 f9 81 c1 68 02 00 00 c1 e9 03 <f3> 48 ab 48 8b 83 30 01 00 00 48 85 c0 74 0c 48 8d 50 18 48 89 50
[1589503.880310] RSP: 0018:ffffb16bc0a5fd60 EFLAGS: 00000212 ORIG_RAX: ffffffffffffffda
[1589503.888058] RAX: 0000000000000000 RBX: ffff8eca27e1f3a8 RCX: 000000000000001f
[1589503.895365] RDX: ffff8eca237c0000 RSI: ffff8eca27e1f3a8 RDI: ffff8eca27e1f660
[1589503.902680] RBP: ffff8eca26dc0780 R08: 0000000000200000 R09: 0000000000000000
[1589503.909996] R10: 000000000000001f R11: ffff8eca2605a100 R12: ffff8eca27e1f4f0
[1589503.917309] R13: ffff8eca26dc2540 R14: ffff8eca24609f00 R15: ffff8eca26dc15a0
[1589503.924710]  drv_add_interface+0x4c/0x130 [mac80211]
[1589503.929955]  ieee80211_reconfig+0x226/0x1420 [mac80211]
[1589503.935369]  ? rcu_exp_wait_wake+0x240/0x240
[1589503.939821]  ? try_to_del_timer_sync+0x4d/0x80
[1589503.944521]  ieee80211_restart_work+0xbb/0xe0 [mac80211]
[1589503.950015]  process_one_work+0x1a7/0x3a0
[1589503.954208]  worker_thread+0x30/0x390
[1589503.958056]  ? create_worker+0x1a0/0x1a0
[1589503.962163]  kthread+0x112/0x130
[1589503.965579]  ? kthread_bind+0x30/0x30
[1589503.969427]  ret_from_fork+0x22/0x40
[1589503.973188] ---[ end trace 90a3696deb151c82 ]---
[1589503.977994] ------------[ cut here ]------------
[1589503.982795] sched: Unexpected reschedule of offline CPU#1!
[1589503.988473] WARNING: CPU: 3 PID: 19198 at arch/x86/kernel/smp.c:128 native_smp_send_reschedule+0x34/0x40
[1589503.998125] Modules linked in: cdc_ether usbnet r8152 mii cmac ctr ccm arc4 8021q garp mrp bridge stp llc wireguard(E) ip6_udp_tunnel udp_tunnel sr_mod cdrom nft_nat amd64_edac_mod ath10k_pci edac_mce_amd kvm_amd ath10k_core kvm evdev nft_fib_inet ath nft_fib_ipv4 nft_fib_ipv6 mac80211 irqbypass nft_fib uas crct10dif_pclmul crc32_pclmul leds_apu cfg80211 usb_storage ghash_clmulni_intel pcspkr sp5100_tco k10temp ccp sg fam15h_power rfkill rng_core nft_masq_ipv4 nft_masq button pcc_cpufreq nft_chain_nat_ipv4 nf_nat_ipv4 acpi_cpufreq nf_nat nft_ct nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 libcrc32c nf_tables_set nf_tables nfnetlink ip_tables x_tables autofs4 ext4 crc16 mbcache jbd2 crc32c_generic fscrypto ecb sd_mod crc32c_intel ahci libahci sdhci_pci libata cqhci sdhci xhci_pci xhci_hcd mmc_core ehci_pci
[1589504.069417]  ehci_hcd scsi_mod aesni_intel igb usbcore aes_x86_64 crypto_simd cryptd glue_helper i2c_piix4 i2c_algo_bit usb_common dca thermal
[1589504.082380] CPU: 3 PID: 19198 Comm: kworker/3:0 Tainted: G      D W   E     4.19.0-18-amd64 #1 Debian 4.19.208-1
[1589504.092721] Hardware name: PC Engines apu2/apu2, BIOS v4.14.0.5 10/11/2021
[1589504.099844] Workqueue: events_freezable ieee80211_restart_work [mac80211]
[1589504.106819] RIP: 0010:native_smp_send_reschedule+0x34/0x40
[1589504.112485] Code: 05 61 58 0e 01 73 15 48 8b 05 b8 93 ea 00 be fd 00 00 00 48 8b 40 30 e9 ea a7 9b 00 89 fe 48 c7 c7 80 d9 a4 b4 e8 95 c3 6b 00 <0f> 0b c3 66 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 8b 05 ed a3 5a
[1589504.131410] RSP: 0018:ffff8eca2ab83698 EFLAGS: 00010082
[1589504.136819] RAX: 0000000000000000 RBX: ffff8eca2aaa2140 RCX: 0000000000000006
[1589504.144134] RDX: 0000000000000007 RSI: 0000000000000092 RDI: ffff8eca2ab966b0
[1589504.151449] RBP: ffff8eca229a8000 R08: 0000000000000579 R09: 000000000000000f
[1589504.158763] R10: 0000000000000000 R11: ffffffffb51fa6ef R12: ffff8eca229a875c
[1589504.166078] R13: ffff8eca2ab836e8 R14: 0000000000000046 R15: 0000000000000001
[1589504.173394] FS:  0000000000000000(0000) GS:ffff8eca2ab80000(0000) knlGS:0000000000000000
[1589504.181661] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[1589504.187589] CR2: 0000000000000fc0 CR3: 00000001291ec000 CR4: 00000000000406e0
[1589504.194903] Call Trace:
[1589504.197538]  <IRQ>
[1589504.199744]  check_preempt_curr+0x7a/0x90
[1589504.203939]  ttwu_do_wakeup+0x19/0x140
[1589504.207874]  try_to_wake_up+0x1d8/0x470
[1589504.211897]  __wake_up_common+0x7a/0x190
[1589504.216010]  ep_poll_callback+0x143/0x2e0
[1589504.220209]  __wake_up_common+0x7a/0x190
[1589504.224325]  __wake_up_common_lock+0x7c/0xc0
[1589504.228779]  ? tick_sched_do_timer+0x60/0x60
[1589504.233232]  irq_work_run_list+0x4d/0x70
[1589504.237340]  update_process_times+0x4a/0x60
[1589504.241707]  tick_sched_handle+0x22/0x60
[1589504.245814]  tick_sched_timer+0x37/0x70
[1589504.249839]  __hrtimer_run_queues+0x100/0x280
[1589504.254382]  hrtimer_interrupt+0x100/0x210
[1589504.258668]  ? up+0x12/0x50
[1589504.261651]  smp_apic_timer_interrupt+0x6a/0x140
[1589504.266450]  apic_timer_interrupt+0xf/0x20
[1589504.270740] RIP: 0010:panic+0x204/0x24a
[1589504.274762] Code: eb a6 83 3d 9d cf ee 00 00 74 05 e8 46 8e 99 ff 48 c7 c6 00 1e 1f b5 48 c7 c7 20 74 a5 b4 e8 91 56 00 00 fb 66 0f 1f 44 00 00 <31> db e8 05 73 a3 ff 4c 39 eb 7c 1d 41 83 f4 01 48 8b 05 45 cf ee
[1589504.293687] RSP: 0018:ffff8eca2ab83ab0 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff13
[1589504.301433] RAX: 0000000000000046 RBX: 0000000000000000 RCX: 0000000000000006
[1589504.308740] RDX: 0000000000000000 RSI: 0000000000000096 RDI: ffff8eca2ab966b0
[1589504.316056] RBP: ffff8eca2ab83b20 R08: 0000000000000516 R09: 000000000000000f
[1589504.323370] R10: 0000000000000000 R11: ffffffffb51fa6ef R12: 0000000000000000
[1589504.330684] R13: 0000000000000000 R14: 0000000000000009 R15: 0000000000000001
[1589504.338003]  ? apic_timer_interrupt+0xa/0x20
[1589504.342465]  oops_end.cold.9+0xc/0x18
[1589504.346314]  no_context+0x1be/0x380
[1589504.349989]  __do_page_fault+0xb2/0x4f0
[1589504.354014]  page_fault+0x1e/0x30
[1589504.357520] RIP: 0010:_raw_spin_lock_bh+0x17/0x30
[1589504.362406] Code: 0f 1f 44 00 00 c3 66 66 2e 0f 1f 84 00 00 00 00 00 66 90 0f 1f 44 00 00 65 81 05 10 e2 cd 4b 00 02 00 00 31 c0 ba 01 00 00 00 <f0> 0f b1 17 85 c0 75 01 c3 89 c6 e8 d9 33 99 ff 66 90 c3 66 0f 1f
[1589504.381340] RSP: 0018:ffff8eca2ab83cc0 EFLAGS: 00010246
[1589504.386748] RAX: 0000000000000000 RBX: 0000000000000fc0 RCX: 0000000000000000
[1589504.394063] RDX: 0000000000000001 RSI: ffff8eca27e1f4c8 RDI: 0000000000000fc0
[1589504.401369] RBP: ffff8eca27e1f3a8 R08: 0000000000000000 R09: 0000000000000000
[1589504.408683] R10: ffff8ec935c7e0f8 R11: 00000000000001ff R12: ffffffffc0b66030
[1589504.415990] R13: 0000000000000000 R14: ffff8eca26dc0ff8 R15: ffff8eca27e1e8c0
[1589504.423343]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589504.429656]  ath10k_wmi_tx_beacons_iter+0x24/0x140 [ath10k_core]
[1589504.435884]  ? ath10k_wmi_cmd_send_nowait+0xa0/0xa0 [ath10k_core]
[1589504.442248]  __iterate_interfaces+0x72/0x110 [mac80211]
[1589504.447746]  ieee80211_iterate_active_interfaces_atomic+0xd/0x10 [mac80211]
[1589504.454924]  ath10k_wmi_op_ep_tx_credits+0x20/0x40 [ath10k_core]
[1589504.461145]  ath10k_htc_process_trailer+0x1af/0x280 [ath10k_core]
[1589504.467456]  ath10k_htc_rx_completion_handler+0x97/0x130 [ath10k_core]
[1589504.474196]  ? ath10k_htc_process_trailer+0x280/0x280 [ath10k_core]
[1589504.480652]  ath10k_pci_process_rx_cb+0x174/0x1b0 [ath10k_pci]
[1589504.486707]  ath10k_ce_per_engine_service+0x93/0x100 [ath10k_core]
[1589504.493109]  ath10k_ce_per_engine_service_any+0x75/0xb0 [ath10k_core]
[1589504.499739]  ath10k_pci_napi_poll+0x44/0x100 [ath10k_pci]
[1589504.505325]  net_rx_action+0x149/0x3b0
[1589504.509268]  __do_softirq+0xde/0x2d8
[1589504.513029]  irq_exit+0xba/0xc0
[1589504.516352]  do_IRQ+0x7f/0xe0
[1589504.519511]  common_interrupt+0xf/0xf
[1589504.523352]  </IRQ>
[1589504.525663] RIP: 0010:ath10k_add_interface+0x79/0x1180 [ath10k_core]
[1589504.532195] Code: 89 e1 31 c0 48 c7 83 48 01 00 00 00 00 00 00 48 83 e7 f8 48 c7 83 a8 03 00 00 00 00 00 00 48 29 f9 81 c1 68 02 00 00 c1 e9 03 <f3> 48 ab 48 8b 83 30 01 00 00 48 85 c0 74 0c 48 8d 50 18 48 89 50
[1589504.551120] RSP: 0018:ffffb16bc0a5fd60 EFLAGS: 00000212 ORIG_RAX: ffffffffffffffda
[1589504.558868] RAX: 0000000000000000 RBX: ffff8eca27e1f3a8 RCX: 000000000000001f
[1589504.566183] RDX: ffff8eca237c0000 RSI: ffff8eca27e1f3a8 RDI: ffff8eca27e1f660
[1589504.573498] RBP: ffff8eca26dc0780 R08: 0000000000200000 R09: 0000000000000000
[1589504.580814] R10: 000000000000001f R11: ffff8eca2605a100 R12: ffff8eca27e1f4f0
[1589504.588128] R13: ffff8eca26dc2540 R14: ffff8eca24609f00 R15: ffff8eca26dc15a0
[1589504.595528]  drv_add_interface+0x4c/0x130 [mac80211]
[1589504.600775]  ieee80211_reconfig+0x226/0x1420 [mac80211]
[1589504.606187]  ? rcu_exp_wait_wake+0x240/0x240
[1589504.610639]  ? try_to_del_timer_sync+0x4d/0x80
[1589504.615338]  ieee80211_restart_work+0xbb/0xe0 [mac80211]
[1589504.620834]  process_one_work+0x1a7/0x3a0
[1589504.625026]  worker_thread+0x30/0x390
[1589504.628874]  ? create_worker+0x1a0/0x1a0
[1589504.632983]  kthread+0x112/0x130
[1589504.636397]  ? kthread_bind+0x30/0x30
[1589504.640245]  ret_from_fork+0x22/0x40
[1589504.644006] ---[ end trace 90a3696deb151c83 ]---

Solutions you've tried The watchdog timer turns this from a major issue into a tolerable one.

miczyg1 commented 2 years ago

@dseomn as per our documentation of mPCIe modules: https://github.com/pcengines/apu2-documentation/blob/master/docs/mpcie_modules.md you may want to:

Then see if it helps

dseomn commented 2 years ago

Thanks for the link, I just added amd_iommu=off. I'll update here if it panics again.

dseomn commented 2 years ago

It panicked again, this time with amd_iommu=off.

metux commented 1 year ago

Null pointer dereference: looks like a kernel bug in ath driver - please report to the corresponding kernel maintainer. In your case - Debian kernel - the right place is the Debain Kernel Team.

Note that HW vendors usually neither have the capacity nor the knowledge to give SW or OS support.