BUG: unable to handle page fault

[plq]

System information

Type	Version/Name
Distribution Name	gentoo
Distribution Version	n/a
Kernel Version	6.1.2-gentoo
Architecture	x86_64
OpenZFS Version	2.1.9

Describe the problem you're observing

all fs operations froze. zfs is not my root partition yet any command that needed storage access froze. rebooted using alt+sysrq+b

Describe how to reproduce the problem

The software I'm working on has a database stress test (sqlite). I was running that at full scale. Could not reproduce this afterwards no matter what

Include any warning/errors/backtraces from the system logs

Mar  9 03:39:42 mint kernel: BUG: unable to handle page fault for address: ffffaea9b0dbe000
Mar  9 03:39:42 mint kernel: #PF: supervisor read access in kernel mode
Mar  9 03:39:42 mint kernel: #PF: error_code(0x0000) - not-present page
Mar  9 03:39:42 mint kernel: PGD 100000067 P4D 100000067 PUD 1001ed067 PMD 2234f0067 PTE 0
Mar  9 03:39:42 mint kernel: Oops: 0000 [#1] PREEMPT SMP NOPTI
Mar  9 03:39:42 mint kernel: CPU: 12 PID: 20602 Comm: z_wr_iss Tainted: P           O       6.1.12-gentoo-x86_64 #6
Mar  9 03:39:42 mint kernel: Hardware name: LENOVO 21CQ000GUS/21CQ000GUS, BIOS R22ET55W (1.25 ) 09/14/2022
Mar  9 03:39:42 mint kernel: RIP: 0010:lz4_compress_zfs+0x62d/0x7c0 [zfs]
Mar  9 03:39:42 mint kernel: Code: 79 01 00 00 41 c6 03 00 49 8d 43 01 48 89 d6 4c 8d 42 04 48 83 c0 02 48 29 ce 4c 89 c2 66 89 70 fe 48 8d 71 04 49 39 f8 73 1c <48> 8b 0e 4c 8b 22 4c 39 e1
 0f 85 34 ff ff ff 48 83 c2 08 48 83 c6
Mar  9 03:39:42 mint kernel: RSP: 0018:ffffaea98e8d3d00 EFLAGS: 00010297
Mar  9 03:39:42 mint kernel: RAX: ffffaea9bf96ab68 RBX: 000000000001b200 RCX: 0000000000000000
Mar  9 03:39:42 mint kernel: RDX: ffffaea9b0dbe000 RSI: ffffaea9b0dbdffe RDI: ffffaea9b0dbf1f4
Mar  9 03:39:42 mint kernel: RBP: ffffaea9bf95c000 R08: ffffaea9b0dbdf28 R09: 000000000000ffff
Mar  9 03:39:42 mint kernel: R10: ffffaea9bf95c004 R11: ffffaea9bf96ab65 R12: 0000000000000000
Mar  9 03:39:42 mint kernel: R13: ffffaea9b0da4000 R14: ffff8c8d9295c000 R15: ffffaea9bf973bc0
Mar  9 03:39:42 mint kernel: FS:  0000000000000000(0000) GS:ffff8c949f100000(0000) knlGS:0000000000000000
Mar  9 03:39:42 mint kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 03:39:42 mint kernel: CR2: ffffaea9b0dbe000 CR3: 000000025743a000 CR4: 0000000000750ee0
Mar  9 03:39:42 mint kernel: PKRU: 55555554
Mar  9 03:39:42 mint kernel: Call Trace:
Mar  9 03:39:42 mint kernel:  <TASK>
Mar  9 03:39:42 mint kernel:  zio_compress_data+0xc4/0x100 [zfs]
Mar  9 03:39:42 mint kernel:  zio_interrupt+0x10c4/0x1c60 [zfs]
Mar  9 03:39:42 mint kernel:  zio_execute+0x83/0xde0 [zfs]
Mar  9 03:39:42 mint kernel:  taskq_dispatch+0x4ac/0x6a0 [spl]
Mar  9 03:39:42 mint kernel:  ? wake_up_q+0x90/0x90
Mar  9 03:39:42 mint kernel:  ? zio_data_buf_free+0x17a0/0x17a0 [zfs]
Mar  9 03:39:42 mint kernel:  ? taskq_dispatch+0x250/0x6a0 [spl]
Mar  9 03:39:42 mint kernel:  kthread+0xda/0x100
Mar  9 03:39:42 mint kernel:  ? kthread_complete_and_exit+0x20/0x20
Mar  9 03:39:42 mint kernel:  ret_from_fork+0x22/0x30
Mar  9 03:39:42 mint kernel:  </TASK>
Mar  9 03:39:42 mint kernel: Modules linked in: hid_logitech_hidpp hid_logitech_dj uinput fuse rfcomm tun michael_mic bnep joydev binfmt_misc qrtr_mhi intel_rapl_msr intel_rapl_common zfs(P
O) edac_mce_amd qrtr zunicode(PO) zzstd(O) amdgpu ath11k_pci snd_ctl_led zlua(O) vboxnetadp(O) ath11k zavl(PO) snd_hda_codec_realtek kvm_amd iommu_v2 qmi_helpers icp(PO) gpu_sched vboxnetfl
t(O) snd_hda_codec_generic snd_hda_codec_hdmi uvcvideo kvm i2c_algo_bit snd_hda_intel drm_buddy videobuf2_vmalloc irqbypass snd_intel_dspcfg drm_ttm_helper btusb videobuf2_memops snd_intel_
sdw_acpi mac80211 ttm zcommon(PO) videobuf2_v4l2 crct10dif_pclmul btrtl snd_hda_codec btbcm ghash_clmulni_intel libarc4 videobuf2_common znvpair(PO) thinkpad_acpi drm_display_helper btintel
 sha512_ssse3 snd_hda_core think_lmi snd_pci_acp5x snd_hwdep ledtrig_audio btmtk hid_multitouch spl(O) wmi_bmof firmware_attributes_class snd_rn_pci_acp3x cec platform_profile vboxdrv(O) ra
pl snd_pcm videodev snd_acp_config cfg80211 bluetooth sp5100_tco pcspkr
Mar  9 03:39:42 mint kernel:  drm_kms_helper video snd_soc_acpi snd_timer ucsi_acpi mc dm_multipath ecdh_generic rfkill serio_raw typec_ucsi k10temp thunderbolt i2c_piix4 mhi drm snd snd_pc
i_acp3x ccp typec soundcore wmi i2c_hid_acpi i2c_hid amd_pmc acpi_cpufreq acpi_tad xfs crc32_pclmul crc32c_intel nvme nvme_core
Mar  9 03:39:42 mint kernel: CR2: ffffaea9b0dbe000
Mar  9 03:39:42 mint kernel: ---[ end trace 0000000000000000 ]---
Mar  9 03:39:42 mint kernel: RIP: 0010:lz4_compress_zfs+0x62d/0x7c0 [zfs]
Mar  9 03:39:42 mint kernel: Code: 79 01 00 00 41 c6 03 00 49 8d 43 01 48 89 d6 4c 8d 42 04 48 83 c0 02 48 29 ce 4c 89 c2 66 89 70 fe 48 8d 71 04 49 39 f8 73 1c <48> 8b 0e 4c 8b 22 4c 39 e1 0f 85 34 ff ff ff 48 83 c2 08 48 83 c6
Mar  9 03:39:42 mint kernel: RSP: 0018:ffffaea98e8d3d00 EFLAGS: 00010297
Mar  9 03:39:42 mint kernel: RAX: ffffaea9bf96ab68 RBX: 000000000001b200 RCX: 0000000000000000
Mar  9 03:39:42 mint kernel: RDX: ffffaea9b0dbe000 RSI: ffffaea9b0dbdffe RDI: ffffaea9b0dbf1f4
Mar  9 03:39:42 mint kernel: RBP: ffffaea9bf95c000 R08: ffffaea9b0dbdf28 R09: 000000000000ffff
Mar  9 03:39:42 mint kernel: R10: ffffaea9bf95c004 R11: ffffaea9bf96ab65 R12: 0000000000000000
Mar  9 03:39:42 mint kernel: R13: ffffaea9b0da4000 R14: ffff8c8d9295c000 R15: ffffaea9bf973bc0
Mar  9 03:39:42 mint kernel: FS:  0000000000000000(0000) GS:ffff8c949f100000(0000) knlGS:0000000000000000
Mar  9 03:39:42 mint kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Mar  9 03:39:42 mint kernel: CR2: ffffaea9b0dbe000 CR3: 000000025743a000 CR4: 0000000000750ee0
Mar  9 03:39:42 mint kernel: PKRU: 55555554

-->

[ryao]

https://www.kernel.org/doc/html/latest/x86/x86_64/mm.html

The memory is in the “direct mapping of all physical memory (page_offset_base)“, but was somehow unmapped. It is possible that this is not our bug such that there was either a bug elsewhere in the kernel or a device somehow did a wild write into the page table. We can rule out a bitflip on the present bit since we would not have a zero pointer in the PTE if that happened.

We can harden the code against bugs in itself by adding assertions to our use of kunmap/kunmap_atomic() to catch instances where memory from that region (or more specifically, memory from any region where kmap()/kmap_atomic() does not return an address) is passed to kunmap()/kunmap_atomic(). Given the expense of mapping/unmapping kernel memory, it might not hurt to make it a VERIFY3P() statement so that the assertion is run on non-debug builds too.

I am preparing for a 2 week trip, so I might not send a patch to harden the code against this class of bugs until I return.

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.