arieleizenberg commented 8 months ago

System information

Ubuntu 22.04 running 6.2.0-1018-aws #18\~22.04.1-Ubuntu. zfs-2.1.5-1ubuntu6~22.04.2 zfs-kmod-2.1.9-2ubuntu1.1 on AWS u-12tb1.112xlarge x86_64 instance.

Describe the problem you're observing

We are getting these message in dmesg:

[883114.230457] usercopy: Kernel memory exposure attempt detected from vmalloc (offset 975108, size 249596)!
[883114.240825] ------------[ cut here ]------------
[883114.240828] kernel BUG at mm/usercopy.c:102!
[883114.244737] invalid opcode: 0000 [#1] SMP NOPTI
[883114.248736] CPU: 87 PID: 3614547 Comm: python Tainted: P           O       6.2.0-1018-aws #18~22.04.1-Ubuntu
[883114.256687] Hardware name: Amazon EC2 u-12tb1.112xlarge/, BIOS 1.0 10/16/2017
[883114.263562] RIP: 0010:usercopy_abort+0x6e/0x80
[883114.267546] Code: 76 9a a8 50 48 c7 c2 42 53 95 a8 57 48 c7 c7 50 c7 96 a8 48 0f 44 d6 48 c7 c6 f3 c7 96 a8 4c 89 d1 49 0f 44 f3 e8 f2 07 d2 ff <0f> 0b 49 c7 c1 a0 58 99 a8 4c 89 cf 4d 89 c8 eb a9 90 90 90 90 90
[883114.280934] RSP: 0018:ffffbc3498033b00 EFLAGS: 00010246
[883114.285235] RAX: 000000000000005c RBX: 0000000000000001 RCX: 0000000000000000
[883114.292197] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[883114.299145] RBP: ffffbc3498033b18 R08: 0000000000000000 R09: 0000000000000000
[883114.306143] R10: 0000000000000000 R11: 0000000000000000 R12: ffffbc3c94605104
[883114.313346] R13: 000000000003cefc R14: 000000000003cefc R15: 0000000000000000
[883114.321434] FS:  00007fde416ff640(0000) GS:ffff990af24c0000(0000) knlGS:0000000000000000
[883114.329637] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[883114.334776] CR2: 00007fde38c8c000 CR3: 00000004a5036005 CR4: 00000000007706e0
[883114.342569] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[883114.350493] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[883114.358410] PKRU: 55555554
[883114.361890] Call Trace:
[883114.365554]  <TASK>
[883114.368916]  ? show_regs+0x72/0x90
[883114.372913]  ? die+0x38/0xb0
[883114.376655]  ? do_trap+0xe3/0x100
[883114.380867]  ? do_error_trap+0x75/0xb0
[883114.384984]  ? usercopy_abort+0x6e/0x80
[883114.389183]  ? exc_invalid_op+0x53/0x80
[883114.393509]  ? usercopy_abort+0x6e/0x80
[883114.397773]  ? asm_exc_invalid_op+0x1b/0x20
[883114.402215]  ? usercopy_abort+0x6e/0x80
[883114.406467]  check_heap_object+0x149/0x1d0
[883114.410559]  __check_object_size.part.0+0x72/0x150
[883114.414874]  __check_object_size+0x23/0x30
[883114.418884]  zfs_uiomove_iter+0x63/0x100 [zfs]
[883114.423666]  zfs_uiomove+0x34/0x60 [zfs]
[883114.427620]  dmu_read_uio_dnode+0xaf/0x110 [zfs]
[883114.431812]  dmu_read_uio_dbuf+0x47/0x70 [zfs]
[883114.435988]  zfs_read+0x13a/0x3e0 [zfs]
[883114.440063]  zpl_iter_read+0xa3/0x110 [zfs]
[883114.444165]  vfs_read+0x219/0x2f0
[883114.447724]  __x64_sys_pread64+0x9e/0xd0
[883114.451556]  do_syscall_64+0x59/0x90
[883114.455217]  ? __rseq_handle_notify_resume+0x2d/0xf0
[883114.459653]  ? exit_to_user_mode_loop+0xf1/0x140
[883114.463971]  ? exit_to_user_mode_prepare+0xaf/0xd0
[883114.468577]  ? irqentry_exit_to_user_mode+0x17/0x20
[883114.472852]  ? irqentry_exit+0x21/0x40
[883114.476599]  ? exc_page_fault+0x92/0x190
[883114.480376]  entry_SYSCALL_64_after_hwframe+0x73/0xdd
[883114.484631] RIP: 0033:0x7fe0b511278f
[883114.488275] Code: 08 89 3c 24 48 89 4c 24 18 e8 7d e2 f7 ff 4c 8b 54 24 18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 04 24 e8 bd e2 f7 ff 48 8b
[883114.501858] RSP: 002b:00007fde416feaa0 EFLAGS: 00000293 ORIG_RAX: 0000000000000011
[883114.509142] RAX: ffffffffffffffda RBX: 00000000002cf224 RCX: 00007fe0b511278f
[883114.516220] RDX: 00000000002cf224 RSI: 00007fde38c88f00 RDI: 0000000000000014
[883114.523184] RBP: 00007fde38c88f00 R08: 0000000000000000 R09: 0000000000400000
[883114.530148] R10: 0000000000000004 R11: 0000000000000293 R12: 0000000000000004
[883114.537162] R13: 0000000000000014 R14: 0000000000000000 R15: 00000000002cf224
[883114.544125]  </TASK>
[883114.547193] Modules linked in: netlink_diag rpcrdma rdma_cm iw_cm ib_cm ib_core nfsd nfs_acl xt_tcpudp rpcsec_gss_krb5 auth_rpcgss nfsv4 nfs lockd grace fscache netfs xt_conntrack nft_chain_nat xt_MASQUERADE nf_nat nf_conntrack_netlink nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 xfrm_user xfrm_algo xt_addrtype nft_compat nf_tables libcrc32c nfnetlink br_netfilter bridge stp llc tls nvme_fabrics overlay sunrpc binfmt_misc intel_rapl_msr intel_rapl_common intel_uncore_frequency_common zfs(PO) zunicode(PO) isst_if_common zzstd(O) zlua(O) nfit zavl(PO) crct10dif_pclmul icp(PO) crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel zcommon(PO) nls_iso8859_1 sha512_ssse3 ppdev znvpair(PO) aesni_intel crypto_simd spl(O) cryptd rapl input_leds psmouse i2c_piix4 ena serio_raw parport_pc parport mac_hid dm_multipath scsi_dh_rdac scsi_dh_emc scsi_dh_alua sch_fq_codel msr drm efi_pstore ip_tables x_tables autofs4
[883114.602863] ---[ end trace 0000000000000000 ]---
[883114.606941] RIP: 0010:usercopy_abort+0x6e/0x80
[883114.611011] Code: 76 9a a8 50 48 c7 c2 42 53 95 a8 57 48 c7 c7 50 c7 96 a8 48 0f 44 d6 48 c7 c6 f3 c7 96 a8 4c 89 d1 49 0f 44 f3 e8 f2 07 d2 ff <0f> 0b 49 c7 c1 a0 58 99 a8 4c 89 cf 4d 89 c8 eb a9 90 90 90 90 90
[883114.624735] RSP: 0018:ffffbc3498033b00 EFLAGS: 00010246
[883114.629096] RAX: 000000000000005c RBX: 0000000000000001 RCX: 0000000000000000
[883114.636129] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[883114.643193] RBP: ffffbc3498033b18 R08: 0000000000000000 R09: 0000000000000000
[883114.650197] R10: 0000000000000000 R11: 0000000000000000 R12: ffffbc3c94605104
[883114.657247] R13: 000000000003cefc R14: 000000000003cefc R15: 0000000000000000
[883114.664244] FS:  00007fde416ff640(0000) GS:ffff990af24c0000(0000) knlGS:0000000000000000
[883114.671730] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[883114.676214] CR2: 00007fde38c8c000 CR3: 00000004a5036005 CR4: 00000000007706e0
[883114.683232] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[883114.690268] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[883114.697324] PKRU: 55555554

rincebrain commented 8 months ago

15668 and #12543 seems germane, perhaps.

That said, I'd suggest you try 2.1.14 or 2.2.2, to be sure it's not some other quirk that's been fixed, and/or you could try the workaround mentioned here.

arieleizenberg commented 8 months ago

Will check and update, thanks!

pando85 commented 7 months ago

Same here:

[Sat Apr 13 11:23:40 2024] usercopy: Kernel memory exposure attempt detected from vmalloc (offset 350176, size 219168)!
[Sat Apr 13 11:23:40 2024] ------------[ cut here ]------------
[Sat Apr 13 11:23:40 2024] kernel BUG at mm/usercopy.c:102!
[Sat Apr 13 11:23:40 2024] invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
[Sat Apr 13 11:23:40 2024] CPU: 15 PID: 16096 Comm: tokio-runtime-w Tainted: P           OE      6.8.0-22-generic #22-Ubuntu
[Sat Apr 13 11:23:40 2024] Hardware name: ASUS System Product Name/PRIME X670-P, BIOS 1811 10/07/2023
[Sat Apr 13 11:23:40 2024] RIP: 0010:usercopy_abort+0x6c/0x80
[Sat Apr 13 11:23:40 2024] Code: 3e 99 51 48 c7 c2 64 bf 40 99 41 52 48 c7 c7 08 4d 48 99 48 0f 45 d6 48 c7 c6 ac 12 3e 99 48 89 c1 49 0f 45 f3 e8 04 a3 cd ff <0f> 0b 49 c7 c1 3a ae 3f 99 4d 89 ca 4d 89 c8 eb a8 0f 1f 00 90 90
[Sat Apr 13 11:23:40 2024] RSP: 0018:ffffb5d87a8dbb28 EFLAGS: 00010246
[Sat Apr 13 11:23:40 2024] RAX: 000000000000005c RBX: ffffb5d8aada27e0 RCX: 0000000000000000
[Sat Apr 13 11:23:40 2024] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
[Sat Apr 13 11:23:40 2024] RBP: ffffb5d87a8dbb40 R08: 0000000000000000 R09: 0000000000000000
[Sat Apr 13 11:23:40 2024] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000035820
[Sat Apr 13 11:23:40 2024] R13: 0000000000000001 R14: ffffb5d8aadd8000 R15: ffffb5d87a8dbd98
[Sat Apr 13 11:23:40 2024] FS:  00007fb8e5a00700(0000) GS:ffff9e1e1df80000(0000) knlGS:0000000000000000
[Sat Apr 13 11:23:40 2024] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[Sat Apr 13 11:23:40 2024] CR2: 00007fb828e17000 CR3: 000000019da86000 CR4: 0000000000f50ef0
[Sat Apr 13 11:23:40 2024] PKRU: 55555554
[Sat Apr 13 11:23:40 2024] Call Trace:
[Sat Apr 13 11:23:40 2024]  <TASK>
[Sat Apr 13 11:23:40 2024]  ? show_regs+0x6d/0x80
[Sat Apr 13 11:23:40 2024]  ? die+0x37/0xa0
[Sat Apr 13 11:23:40 2024]  ? do_trap+0xd4/0xf0
[Sat Apr 13 11:23:40 2024]  ? do_error_trap+0x71/0xb0
[Sat Apr 13 11:23:40 2024]  ? usercopy_abort+0x6c/0x80
[Sat Apr 13 11:23:40 2024]  ? exc_invalid_op+0x52/0x80
[Sat Apr 13 11:23:40 2024]  ? usercopy_abort+0x6c/0x80
[Sat Apr 13 11:23:40 2024]  ? asm_exc_invalid_op+0x1b/0x20
[Sat Apr 13 11:23:40 2024]  ? usercopy_abort+0x6c/0x80
[Sat Apr 13 11:23:40 2024]  ? usercopy_abort+0x6c/0x80
[Sat Apr 13 11:23:40 2024]  check_heap_object+0x14c/0x1e0
[Sat Apr 13 11:23:40 2024]  __check_object_size.part.0+0x72/0x150
[Sat Apr 13 11:23:40 2024]  __check_object_size+0x23/0x30
[Sat Apr 13 11:23:40 2024]  zfs_uiomove_iter+0x60/0x100 [zfs]
[Sat Apr 13 11:23:40 2024]  zfs_uiomove+0x34/0x60 [zfs]
[Sat Apr 13 11:23:40 2024]  dmu_read_uio_dnode+0xbb/0x110 [zfs]
[Sat Apr 13 11:23:40 2024]  dmu_read_uio_dbuf+0x48/0x70 [zfs]
[Sat Apr 13 11:23:40 2024]  zfs_read+0x125/0x300 [zfs]
[Sat Apr 13 11:23:40 2024]  zpl_iter_read+0xbc/0x130 [zfs]
[Sat Apr 13 11:23:40 2024]  vfs_read+0x255/0x390
[Sat Apr 13 11:23:40 2024]  ksys_read+0x73/0x100
[Sat Apr 13 11:23:40 2024]  __x64_sys_read+0x19/0x30
[Sat Apr 13 11:23:40 2024]  do_syscall_64+0x82/0x180
[Sat Apr 13 11:23:40 2024]  ? count_memcg_events.constprop.0+0x2a/0x50
[Sat Apr 13 11:23:40 2024]  ? srso_alias_return_thunk+0x5/0xfbef5
[Sat Apr 13 11:23:40 2024]  ? handle_mm_fault+0xad/0x380
[Sat Apr 13 11:23:40 2024]  ? srso_alias_return_thunk+0x5/0xfbef5
[Sat Apr 13 11:23:40 2024]  ? do_user_addr_fault+0x338/0x6b0
[Sat Apr 13 11:23:40 2024]  ? srso_alias_return_thunk+0x5/0xfbef5
[Sat Apr 13 11:23:40 2024]  ? irqentry_exit_to_user_mode+0x7b/0x260
[Sat Apr 13 11:23:40 2024]  ? srso_alias_return_thunk+0x5/0xfbef5
[Sat Apr 13 11:23:40 2024]  ? irqentry_exit+0x43/0x50
[Sat Apr 13 11:23:40 2024]  ? srso_alias_return_thunk+0x5/0xfbef5
[Sat Apr 13 11:23:40 2024]  ? exc_page_fault+0x94/0x1b0
[Sat Apr 13 11:23:40 2024]  entry_SYSCALL_64_after_hwframe+0x6e/0x76
[Sat Apr 13 11:23:40 2024] RIP: 0033:0x7fb8fb000544
[Sat Apr 13 11:23:40 2024] Code: Unable to access opcode bytes at 0x7fb8fb00051a.
[Sat Apr 13 11:23:40 2024] RSP: 002b:00007fb8e59fa710 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[Sat Apr 13 11:23:40 2024] RAX: ffffffffffffffda RBX: 000000000000003c RCX: 00007fb8fb000544
[Sat Apr 13 11:23:40 2024] RDX: 0000000000f3dd4d RSI: 00007fb82898c820 RDI: 000000000000003c
[Sat Apr 13 11:23:40 2024] RBP: 00007fb82898c820 R08: 0000000000000000 R09: 00007fb828000000
[Sat Apr 13 11:23:40 2024] R10: fffffffffffff000 R11: 0000000000000246 R12: 0000000000f3dd4d
[Sat Apr 13 11:23:40 2024] R13: 7fffffffffffffff R14: 0000000000f40000 R15: 00007fb8e59fa810
[Sat Apr 13 11:23:40 2024]  </TASK>
[Sat Apr 13 11:23:40 2024] Modules linked in: tls xt_TPROXY nf_tproxy_ipv6 nf_tproxy_ipv4 xt_CT cls_bpf sch_ingress vxlan ip6_udp_tunnel udp_tunnel veth xt_socket nf_socket_ipv4 nf_socket_ipv6 ip6table_raw iptable_raw xfrm_user xfrm_algo nf_tables nf_conntrack_netlink xt_statistic xt_nat xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_physdev xt_NFLOG nfnetlink_log xt_limit xt_tcpudp xt_set ip_set_hash_ip xt_mark xt_multiport xt_addrtype ip6table_filter ip6table_nat ip6table_mangle ip6_tables xt_conntrack xt_comment iptable_mangle ip_set iptable_filter iptable_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 overlay br_netfilter bridge stp llc binfmt_misc nvidia_uvm(POE) nvidia_drm(POE) nvidia_modeset(POE) amdgpu snd_hda_codec_realtek snd_hda_codec_generic intel_rapl_msr snd_hda_codec_hdmi intel_rapl_common amd64_edac edac_mce_amd amdxcp drm_exec gpu_sched snd_hda_intel drm_buddy kvm_amd snd_intel_dspcfg drm_suballoc_helper snd_intel_sdw_acpi drm_ttm_helper nls_iso8859_1 nvidia(POE) snd_hda_codec zfs(PO) ttm mfd_aaeon eeepc_wmi
[Sat Apr 13 11:23:40 2024]  snd_hda_core kvm drm_display_helper asus_wmi snd_hwdep spl(O) snd_pcm cec ledtrig_audio irqbypass snd_timer rc_core sparse_keymap snd rapl i2c_algo_bit ccp wmi_bmof i2c_piix4 k10temp platform_profile soundcore gpio_amdpt joydev input_leds mac_hid cfg80211 dm_multipath efi_pstore nfnetlink dmi_sysfs ip_tables x_tables autofs4 btrfs blake2b_generic raid10 raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq libcrc32c raid1 raid0 hid_generic uas usbhid usb_storage hid crct10dif_pclmul crc32_pclmul polyval_clmulni polyval_generic ghash_clmulni_intel sha256_ssse3 sha1_ssse3 nvme r8169 ahci nvme_core xhci_pci realtek libahci xhci_pci_renesas nvme_auth video wmi aesni_intel crypto_simd cryptd
[Sat Apr 13 11:23:40 2024] ---[ end trace 0000000000000000 ]---
[Sat Apr 13 11:23:40 2024] usercopy: Kernel memory exposure attempt detected from vmalloc (offset 995312, size 229392)!

I'm running:

root@prusik:~# zfs version
zfs-2.2.2-0ubuntu8
zfs-kmod-2.2.2-0ubuntu4
root@prusik:~# uname -a
Linux prusik 6.8.0-22-generic #22-Ubuntu SMP PREEMPT_DYNAMIC Thu Apr  4 22:30:32 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
root@prusik:~# cat /etc/os-release 
PRETTY_NAME="Ubuntu Noble Numbat (development branch)"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

pando85 commented 7 months ago

Interesting facts:

My kernel is not in the supported list.
The error appears after an OOM Kill of a process.
If the OOM Kill doesn't happen the error doesn't occur.
ZFS has enough memory because this is happening in a process limited by cgroups.
That process it is doing some operations in the ZFS filesystem and it can be that has some threads or forks.

zhanglei002 commented 6 months ago

similar issue here:

[148683.969299] usercopy: Kernel memory overwrite attempt detected to vmalloc 'no area' (offset 0, size 159744)!
[148683.969387] ------------[ cut here ]------------
[148683.969388] kernel BUG at mm/usercopy.c:102!
[148683.969393] invalid opcode: 0000 [#1] PREEMPT SMP PTI
[148683.969406] CPU: 2 PID: 476341 Comm: kworker/u8:0 Tainted: P           OE      6.8.6-1-MANJARO #1 b41004827636010039b162382595c97c32aa6312
[148683.969427] Hardware name: ASUSTeK COMPUTER INC. X550VL/X550VL, BIOS X550VL.205 02/28/2014
[148683.969442] Workqueue: loop0 loop_workfn [loop]
[148683.969460] RIP: 0010:usercopy_abort+0x6c/0x80
[148683.969473] Code: 3d a9 51 48 c7 c2 6c e0 3f a9 41 52 48 c7 c7 78 79 46 a9 48 0f 45 d6 48 c7 c6 fc 77 3d a9 48 89 c1 49 0f 45 f3 e8 24 e9 d3 ff <0f> 0b 49 c7 c1 69 65 3d a9 4d 89 ca 4d 89 c8 eb a8 0f 1f 00 90 90
[148683.969500] RSP: 0018:ffffb270984579c0 EFLAGS: 00010246
[148683.969512] RAX: 0000000000000060 RBX: ffffb2709c027000 RCX: 0000000000000000
[148683.969525] RDX: 0000000000000000 RSI: ffff9f48e2f219c0 RDI: ffff9f48e2f219c0
[148683.969537] RBP: 0000000000027000 R08: 0000000000000000 R09: ffffb27098457868
[148683.969550] R10: ffffb27098457860 R11: 0000000000000003 R12: 0000000000000000
[148683.969562] R13: ffffb2709c04e000 R14: 0000000000000001 R15: ffffb27098457d18
[148683.969575] FS:  0000000000000000(0000) GS:ffff9f48e2f00000(0000) knlGS:0000000000000000
[148683.969589] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[148683.969600] CR2: 0000000064202c0c CR3: 0000000188e20005 CR4: 00000000001726f0
[148683.969613] Call Trace:
[148683.969621]  <TASK>
[148683.969629]  ? die+0x36/0x90
[148683.969639]  ? do_trap+0xda/0x100
[148683.969652]  ? usercopy_abort+0x6c/0x80
[148683.969663]  ? do_error_trap+0x6a/0x90
[148683.969673]  ? usercopy_abort+0x6c/0x80
[148683.969683]  ? exc_invalid_op+0x50/0x70
[148683.969694]  ? usercopy_abort+0x6c/0x80
[148683.969704]  ? asm_exc_invalid_op+0x1a/0x20
[148683.969715]  ? usercopy_abort+0x6c/0x80
[148683.969725]  __check_object_size+0x2b1/0x2c0
[148683.969736]  zfs_uiomove_iter+0xa5/0xe0 [zfs aa5285c4094032c116f2930c685fb0240b8a4412]
[148683.969953]  dmu_write_uio_dnode+0xc1/0x1d0 [zfs aa5285c4094032c116f2930c685fb0240b8a4412]
[148683.970150]  dmu_write_uio_dbuf+0x4e/0x70 [zfs aa5285c4094032c116f2930c685fb0240b8a4412]
[148683.970345]  zfs_write+0x4ea/0xc70 [zfs aa5285c4094032c116f2930c685fb0240b8a4412]
[148683.970541]  zpl_iter_write+0x113/0x190 [zfs aa5285c4094032c116f2930c685fb0240b8a4412]
[148683.970722]  lo_rw_aio.isra.0+0x29d/0x2b0 [loop 17306f0c8a0020772d2591466362b74ef771094a]
[148683.970741]  ? sched_clock+0x10/0x30
[148683.970752]  loop_process_work+0xb2/0x960 [loop 17306f0c8a0020772d2591466362b74ef771094a]
[148683.970770]  ? finish_task_switch.isra.0+0x94/0x2f0
[148683.970784]  ? __schedule+0x3ee/0x1520
[148683.970795]  process_one_work+0x17b/0x350
[148683.970806]  worker_thread+0x30f/0x450
[148683.970816]  ? __pfx_worker_thread+0x10/0x10
[148683.970826]  kthread+0xe8/0x120
[148683.970837]  ? __pfx_kthread+0x10/0x10
[148683.971333]  ret_from_fork+0x34/0x50
[148683.971799]  ? __pfx_kthread+0x10/0x10
[148683.972262]  ret_from_fork_asm+0x1b/0x30
[148683.972731]  </TASK>

zfs version:

zfs-2.2.3-1
zfs-kmod-2.2.3-1

kernel:

6.8.6-1-MANJARO #1 SMP PREEMPT_DYNAMIC Sat Apr 13 15:48:36 UTC 2024 x86_64 GNU/Linux

amotin commented 6 months ago

@ixhamza guessed, and I tend to agree, that it may be related to issue fixed by #16042 . We just not sure what can be the trigger here, if you have block cloning disabled as in 2.2.3 by default.

robn commented 6 months ago

I can't reproduce it (yet), but I've been reading code. On the kernel side, the two errors come from this part of check_heap_object():

    if (is_vmalloc_addr(ptr) && !pagefault_disabled()) {
        struct vmap_area *area = find_vmap_area(addr);

        if (!area)
            usercopy_abort("vmalloc", "no area", to_user, 0, n);

        if (n > area->va_end - addr) {
            offset = addr - area->va_start;
            usercopy_abort("vmalloc", NULL, to_user, offset, n);
        }
        return;
    }

That is, the pointer is partially or entirely outside a valid virtual allocation. So it's effectively a use-after-free, or perhaps, a failed allocation. Given proximity to OOM makes me wonder if another thread freed something in response to a reclaim request, and by the time we get here its gone.

If nothing else, we could use similar checking logic as above ourselves, and if we hit it, log something and return EFAULT? Might at least stop us dying, might even work, and hopefully logging something could get us more information.

openzfs / zfs

ZFS generating "kernel BUG at mm/usercopy.c:102!" #15918

System information

Describe the problem you're observing

15668 and #12543 seems germane, perhaps.