Kernel panic while unloading v23.11 oct_ep_phc driver module from debian kernel

[cbf123]

We have recently seen a repeatable issue where the v23.11 oct_ep_phc driver kernel panics when it is being unloaded. The versioning looks a bit odd, but this is the realtime 5.10.205 kernel imported via the Yocto project. The panic log is below, I've also attached the kernel config file in case it's useful.

[13319.084315] octeon_ep_vf: Unloading ... [13319.084370] octeon_ep_vf: Unloading complete [13320.222135] octeon_ep: Unloading ... [13320.222152] octeon_ep 0000:51:00.0: Removing device. [13320.238273] octeon_ep 0000:51:00.0 enp81s0f0: Stopping the device ... [13320.298436] octeon_ep 0000:51:00.0 enp81s0f0: IRQs freed [13320.298850] octeon_ep 0000:51:00.0: Disabled MSI-X [13320.298856] octeon_ep 0000:51:00.0 enp81s0f0: Freed IOQ Vectors [13320.299849] octeon_ep 0000:51:00.0 enp81s0f0: Device stopped !! [13320.338150] octeon_ep 0000:51:00.0: Cleaning up Octeon Device ... [13320.338153] octeon_ep 0000:51:00.0: Sending dev_unload msg to fw [13320.338157] Octep ctrl mbox : Uninit successful. [13320.338159] octeon_ep 0000:51:00.0: CNXKXX: Doing soft reset [13320.348305] octeon_ep: Unloading complete [13372.984738] Octeon EP PHC 0000:51:00.1: OCT_PHC[0]: Stopping octeon device [13372.984785] BUG: unable to handle page fault for address: 0000000000483eb0 [13376.023616] #PF: supervisor write access in kernel mode [13376.028843] #PF: error_code(0x0002) - not-present page [13376.033981] PGD 1ee41d067 P4D 0 [13376.037214] Oops: 0002 [#1] PREEMPT_RT SMP NOPTI [13376.041833] CPU: 47 PID: 100726 Comm: modprobe Kdump: loaded Tainted: G S O 5.10.0-6-rt-amd64 #1 Debian 5.10.205-1.stx.76 [13376.053906] Hardware name: Dell Inc. PowerEdge XR11/07GGDG, BIOS 1.10.2 03/03/2023 [13376.061472] RIP: 0010:native_queued_spin_lock_slowpath+0x19f/0x1e0 [13376.067651] Code: ff ff ff c6 47 01 00 e9 1d ff ff ff c1 ee 12 83 e0 03 83 ee 01 48 c1 e0 05 48 63 f6 48 05 c0 01 03 00 48 03 04 f5 e0 36 9d 86 <48> ... [13376.086397] RSP: 0018:ff55572f07ccbd18 EFLAGS: 00010002 [13376.091623] RAX: 0000000000483eb0 RBX: ff1b649cb058a818 RCX: 0000000000c00000 [13376.098756] RDX: ff1b64baff1f01c0 RSI: 0000000000000c5b RDI: ff1b649cb058a818 [13376.105888] RBP: ff1b649cb058a818 R08: 0000000000c00000 R09: ff1b649c80033e40 [13376.113022] R10: 0000000000000247 R11: fffffffffff9ca84 R12: 0000000000000246 [13376.120155] R13: 0000000000000000 R14: 0000000000000001 R15: ffffffffc0e97210 [13376.127287] FS: 00007facb457f540(0000) GS:ff1b64baff1c0000(0000) knlGS:0000000000000000 [13376.135371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13376.141118] CR2: 0000000000483eb0 CR3: 00000001d2656006 CR4: 0000000000771ee0 [13376.148253] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13376.155386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13376.162518] PKRU: 55555554 [13376.165229] Call Trace: [13376.167686] ? die+0x5d/0xa0 [13376.170742] ? no_context+0x189/0x3a0 [13376.174409] ? exc_page_fault+0x2a1/0x530 [13376.178421] ? dequeue_entity+0xc6/0x4a0 [13376.182348] ? newidle_balance+0x3fd/0x480 [13376.186448] ? asm_exc_page_fault+0x1e/0x30 [13376.190633] ? native_queued_spin_lock_slowpath+0x19f/0x1e0 [13376.196204] _raw_spin_lock_irqsave+0x30/0x40 [13376.200563] rt_spin_lock_slowlock+0x40/0x80 [13376.204837] ? wait_for_completion+0xa0/0xe0 [13376.209109] rt_spin_lock+0x2a/0x40 [13376.212604] wake_up_common_lock+0x60/0xb0 [13376.216876] ptp_clock_unregister+0x2b/0x70 [13376.221062] octeon_ep_phc_remove+0xcc/0x134 [oct_ep_phc] [13376.226461] ? pci_device_remove+0x38/0xa0 [13376.230560] ? device_release_driver+0x17b/0x250 [13376.235353] ? driver_detach+0xc9/0x110 [13376.239192] ? bus_remove_driver+0x5b/0xe0 [13376.243293] ? pci_unregister_driver+0x2a/0xb0 [13376.247736] ? do_sys_delete_module.constprop.0+0x171/0x2c0 [13376.253484] ? vtime_user_exit+0x1c/0x70 [13376.257410] ? __context_tracking_exit+0x56/0xe0 [13376.262028] ? do_syscall_64+0x30/0x40 [13376.265782] ? entry_SYSCALL_64_after_hwframe+0x62/0xc7 [13376.271008] Modules linked in: nft_counter xt_addrtype nft_compat nf_tables br_netfilter ... [13376.271054] acpi_power_meter [last unloaded: octeon_ep] [13376.363222] CR2: 0000000000483eb0

config-5.10.0-6-rt-amd64.txt

[GaniHaseeb]

Hi Chris,

What version is the PHC driver? There was a fix in v23_10

“octeon_device_t pointer was used as a dangling pointer in the remove sequence. Also remove sequence, when firmware ready work occurred, didn't clean resources allocated. Fixed the above bugs.”

Regards, Haseeb

From: Chris Friesen @.> Sent: Thursday, February 8, 2024 8:36 AM To: MarvellEmbeddedProcessors/pcie_ep_octeon_host @.> Cc: Subscribed @.***> Subject: [EXT] [MarvellEmbeddedProcessors/pcie_ep_octeon_host] Kernel panic unloading the v23.11 VF module from debian kernel (Issue #2)

External Email

---

We have recently seen a repeatable issue where the VF driver kernel panics when trying to unload it. The versioning looks a bit odd, but this is the realtime 5.10.205 kernel imported via the Yocto project. The panic log is below, I've also attached the kernel config file in case it's useful.

[13319.084315] octeon_ep_vf: Unloading ... [13319.084370] octeon_ep_vf: Unloading complete [13320.222135] octeon_ep: Unloading ... [13320.222152] octeon_ep 0000:51:00.0: Removing device. [13320.238273] octeon_ep 0000:51:00.0 enp81s0f0: Stopping the device ... [13320.298436] octeon_ep 0000:51:00.0 enp81s0f0: IRQs freed [13320.298850] octeon_ep 0000:51:00.0: Disabled MSI-X [13320.298856] octeon_ep 0000:51:00.0 enp81s0f0: Freed IOQ Vectors [13320.299849] octeon_ep 0000:51:00.0 enp81s0f0: Device stopped !! [13320.338150] octeon_ep 0000:51:00.0: Cleaning up Octeon Device ... [13320.338153] octeon_ep 0000:51:00.0: Sending dev_unload msg to fw [13320.338157] Octep ctrl mbox : Uninit successful. [13320.338159] octeon_ep 0000:51:00.0: CNXKXX: Doing soft reset [13320.348305] octeon_ep: Unloading complete [13372.984738] Octeon EP PHC 0000:51:00.1: OCT_PHC[0]: Stopping octeon device [13372.984785] BUG: unable to handle page fault for address: 0000000000483eb0 [13376.023616] #PF: supervisor write access in kernel mode [13376.028843] #PF: error_code(0x0002) - not-present page [13376.033981] PGD 1ee41d067 P4D 0 [13376.037214] Oops: 0002 [#1https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MarvellEmbeddedProcessors_pcie-5Fep-5Focteon-5Fhost_pull_1&d=DwMCaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=BI2U2eD4pszU042zSUYQkW1tlOcpzMe2L5StjIwOenQ&m=x9kiTKVs8aex7VlyDHHFnW-VupptnyEQE1u5Mk3JmTI1QHc4DutYP93RafGRu9Ao&s=OEiGwMsAQ3Gu_QI9SM9-kQDOLRcVYCd1TRgjbhqZB8w&e=] PREEMPT_RT SMP NOPTI [13376.041833] CPU: 47 PID: 100726 Comm: modprobe Kdump: loaded Tainted: G S O 5.10.0-6-rt-amd64 #1https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MarvellEmbeddedProcessors_pcie-5Fep-5Focteon-5Fhost_pull_1&d=DwMCaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=BI2U2eD4pszU042zSUYQkW1tlOcpzMe2L5StjIwOenQ&m=x9kiTKVs8aex7VlyDHHFnW-VupptnyEQE1u5Mk3JmTI1QHc4DutYP93RafGRu9Ao&s=OEiGwMsAQ3Gu_QI9SM9-kQDOLRcVYCd1TRgjbhqZB8w&e= Debian 5.10.205-1.stx.76 [13376.053906] Hardware name: Dell Inc. PowerEdge XR11/07GGDG, BIOS 1.10.2 03/03/2023 [13376.061472] RIP: 0010:native_queued_spin_lock_slowpath+0x19f/0x1e0 [13376.067651] Code: ff ff ff c6 47 01 00 e9 1d ff ff ff c1 ee 12 83 e0 03 83 ee 01 48 c1 e0 05 48 63 f6 48 05 c0 01 03 00 48 03 04 f5 e0 36 9d 86 <48> ... [13376.086397] RSP: 0018:ff55572f07ccbd18 EFLAGS: 00010002 [13376.091623] RAX: 0000000000483eb0 RBX: ff1b649cb058a818 RCX: 0000000000c00000 [13376.098756] RDX: ff1b64baff1f01c0 RSI: 0000000000000c5b RDI: ff1b649cb058a818 [13376.105888] RBP: ff1b649cb058a818 R08: 0000000000c00000 R09: ff1b649c80033e40 [13376.113022] R10: 0000000000000247 R11: fffffffffff9ca84 R12: 0000000000000246 [13376.120155] R13: 0000000000000000 R14: 0000000000000001 R15: ffffffffc0e97210 [13376.127287] FS: 00007facb457f540(0000) GS:ff1b64baff1c0000(0000) knlGS:0000000000000000 [13376.135371] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [13376.141118] CR2: 0000000000483eb0 CR3: 00000001d2656006 CR4: 0000000000771ee0 [13376.148253] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [13376.155386] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [13376.162518] PKRU: 55555554 [13376.165229] Call Trace: [13376.167686] ? die+0x5d/0xa0 [13376.170742] ? no_context+0x189/0x3a0 [13376.174409] ? exc_page_fault+0x2a1/0x530 [13376.178421] ? dequeue_entity+0xc6/0x4a0 [13376.182348] ? newidle_balance+0x3fd/0x480 [13376.186448] ? asm_exc_page_fault+0x1e/0x30 [13376.190633] ? native_queued_spin_lock_slowpath+0x19f/0x1e0 [13376.196204] _raw_spin_lock_irqsave+0x30/0x40 [13376.200563] rt_spin_lock_slowlock+0x40/0x80 [13376.204837] ? wait_for_completion+0xa0/0xe0 [13376.209109] rt_spin_lock+0x2a/0x40 [13376.212604] wake_up_common_lock+0x60/0xb0 [13376.216876] ptp_clock_unregister+0x2b/0x70 [13376.221062] octeon_ep_phc_remove+0xcc/0x134 [oct_ep_phc] [13376.226461] ? pci_device_remove+0x38/0xa0 [13376.230560] ? device_release_driver+0x17b/0x250 [13376.235353] ? driver_detach+0xc9/0x110 [13376.239192] ? bus_remove_driver+0x5b/0xe0 [13376.243293] ? pci_unregister_driver+0x2a/0xb0 [13376.247736] ? do_sys_delete_module.constprop.0+0x171/0x2c0 [13376.253484] ? vtime_user_exit+0x1c/0x70 [13376.257410] ? __context_tracking_exit+0x56/0xe0 [13376.262028] ? do_syscall_64+0x30/0x40 [13376.265782] ? entry_SYSCALL_64_after_hwframe+0x62/0xc7 [13376.271008] Modules linked in: nft_counter xt_addrtype nft_compat nf_tables br_netfilter ... [13376.271054] acpi_power_meter [last unloaded: octeon_ep] [13376.363222] CR2: 0000000000483eb0

config-5.10.0-6-rt-amd64.txthttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MarvellEmbeddedProcessors_pcie-5Fep-5Focteon-5Fhost_files_14211991_config-2D5.10.0-2D6-2Drt-2Damd64.txt&d=DwMCaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=BI2U2eD4pszU042zSUYQkW1tlOcpzMe2L5StjIwOenQ&m=x9kiTKVs8aex7VlyDHHFnW-VupptnyEQE1u5Mk3JmTI1QHc4DutYP93RafGRu9Ao&s=iUVXmksRuf1gdsZyePCxX62Vmi4m8QBOu0qrolTKoNk&e=

— Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MarvellEmbeddedProcessors_pcie-5Fep-5Focteon-5Fhost_issues_2&d=DwMCaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=BI2U2eD4pszU042zSUYQkW1tlOcpzMe2L5StjIwOenQ&m=x9kiTKVs8aex7VlyDHHFnW-VupptnyEQE1u5Mk3JmTI1QHc4DutYP93RafGRu9Ao&s=gTcuNCFyorWlEjUDJXxuZTgI3sLJqDyEr4SN7MckE3M&e=, or unsubscribehttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BCDJ6BXRGLF6XIS55HYFTLDYST5F3AVCNFSM6AAAAABDABZZGOVHI2DSMVQWIX3LMV43ASLTON2WKOZSGEZDKNRRGE3TSNI&d=DwMCaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=BI2U2eD4pszU042zSUYQkW1tlOcpzMe2L5StjIwOenQ&m=x9kiTKVs8aex7VlyDHHFnW-VupptnyEQE1u5Mk3JmTI1QHc4DutYP93RafGRu9Ao&s=cwhUjd_Q3eDIIOTfz_gLAheOHlQlQ2NH_yME1ZnX4zI&e=. You are receiving this because you are subscribed to this thread.Message ID: @.***>

[cbf123]

I didn't do the test personally, but the developer who did the testing says that the oct_ep_phc driver was also built from the v23.11 tagged release.

The changes in https://github.com/MarvellEmbeddedProcessors/pcie_ep_octeon_host/commit/a0dd5f4fe49f9097c3cd9678b468147e4ffc6ac3 move the kfree() and octeon_free_device_mem() calls down into the "before_exit" clause, but all of that occurs after the call to ptp_clock_unregister() which is implicated in the panic. As such, I don't think that change would have any impact on the panic that was seen.

[GaniHaseeb]

Yes, you are right. The issue seems to happen due to page_fault while holding a spin_lock. One more point, I assume you see this issue only with the RT patch, right?

[m-v-b]

@GaniHaseeb, sorry for the delayed response. I am the person that @cbf123 was referring to with respect to carrying out the driver's testing.

While the bug report is valid, it appears to be a corner case involving the incomplete initialization of the driver when an unsupported device ID is encountered. I apologize for not realizing this before @cbf123 reported this issue.

Please see the initialization logs of the driver below, from the same session that ended with the kernel crash whose logs @cbf123 provided above:

[   13.078037] systemd[1]: Starting Coldplug All udev Devices...
[   13.093017] systemd[1]: Started Journal Service.
[   13.101485] svcrdma: svcrdma is obsoleted, loading rpcrdma instead
[   13.102785] xprtrdma: xprtrdma is obsoleted, loading rpcrdma instead
[   13.108995] Octeon EP PHC 0000:51:00.1: OCT_PHC: Loading PHC driver
[   13.109291] Octeon EP PHC 0000:51:00.1: OCT_PHC[0]:fw ready status 1
[   13.109349] Octeon EP PHC 0000:51:00.1: OCT_PHC: Unknown device found (subsystem_id: 741028)
[   13.109350] Octeon EP PHC 0000:51:00.1: OCT_PHC[0]: Freeing PCI mapped regions for Bar0
[   13.109352] Octeon EP PHC 0000:51:00.1: OCT_PHC[0]: Freeing PCI mapped regions for Bar1
[   13.109353] Octeon EP PHC 0000:51:00.1: OCT_PHC[0]: Freeing PCI mapped regions for Bar2
[   13.109354] Octeon EP PHC 0000:51:00.1: OCT_PHC[0]: Chip specific setup failed
[   13.109355] Octeon EP PHC 0000:51:00.1: OCT_PHC[0]: ERROR: Octeon driver failed to load.
[   13.117400] sctp: Hash tables configured (bind 512/585)
[   13.119575] VFIO - User Level meta-driver version: 0.3

Cross-referencing this with the source code, I observe that when the call to the "octeon_device_init" function fails, the "octeon_device_init_work" function will return (bail out) early. Link: https://github.com/MarvellEmbeddedProcessors/pcie_ep_octeon_host/blob/055d87054688d9620b8b1f48f5ba7a1dd9bbcbd5/drivers/phc/ep_phc.c#L414

This in turns causes the following line to not be executed in such cases:

oct_dev->oct_ep_ptp_clock->ptp_clock = ptp_clock_register(&oct_dev->oct_ep_ptp_clock->caps, NULL);

Because (along with that line) the rest of the "octeon_device_init_work" function is not executed, parts of the octeon_device_t data structure will not be initialized/populated either.

This in turn causes the crash discussed in this issue, as during driver module removal time, the "ptp_clock_unregister" function called by "octeon_ep_phc_remove" will be working with uninitialized memory.

Please note that I did not try using a standard (i.e., non-PREEMPT_RT) kernel, but the nature of the issue makes it obvious that this issue is likely not related to the PREEMPT_RT kernel.

I hope that this helps!

[GaniHaseeb]

Thanks for reporting the issue. We will fix the error path in upcoming release.