search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.64k stars 1.75k forks source link

Setting spl_kmem_cache_slab_limit higher than 16384 makes the magic smoke come out #12983

Open rincebrain opened 2 years ago

rincebrain commented 2 years ago

System information

Type Version/Name
Distribution Name Debian
Distribution Version 11.2
Kernel Version 5.15.11
Architecture x86_64
OpenZFS Version 18168da72

Describe the problem you're observing

(I'll go look into this myself after filing, I just didn't want to forget, since I'm not digging into it immediately...) Curious to know where the optima were, I tried raising spl_kmem_cache_slab_limit past 16384, only to find that it causes one of two kinds of failure.

On the smaller end of values (32768 <= X <= ???), it NULL dereferences on unloading the zfs module, crashing the whole system. On the larger end of values (16777216 being one example), it VERIFY trips immediately on loading the zfs module.

Describe how to reproduce the problem

modprobe spl spl_kmem_cache_slab_limit=32768;modprobe zfs;rmmod zfs;

Include any warning/errors/backtraces from the system logs

From the small end:

[ 5966.247025] BUG: kernel NULL pointer dereference, address: 0000000000000000
[ 5966.247041] #PF: supervisor read access in kernel mode
[ 5966.247049] #PF: error_code(0x0000) - not-present page
[ 5966.247058] PGD 0 P4D 0
[ 5966.247064] Oops: 0000 [#1] SMP NOPTI
[ 5966.247072] CPU: 3 PID: 289714 Comm: rmmod Kdump: loaded Tainted: P           OE     5.15.11badidea1 #1
[ 5966.247087] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[ 5966.247099] RIP: 0010:spl_kmem_cache_destroy+0x28/0x480 [spl]
[ 5966.247113] Code: 81 cf 0f 1f 44 00 00 41 56 41 55 41 54 55 48 89 fd 48 83 ec 48 65 48 8b 04 25 28 00 00 00 48 89 44 24 40 31 c0 48 8d 44 24 08 <81> 3f 2c 2c 2c 2c c7 04 24 00 00 00 00 48 89 44 24 08 48 89 44 24
[ 5966.247140] RSP: 0018:ffff9d82c1d77e18 EFLAGS: 00010246
[ 5966.247149] RAX: ffff9d82c1d77e20 RBX: 0000000000000000 RCX: 000000008010000e
[ 5966.247160] RDX: 000000008010000f RSI: 000000008010000e RDI: 0000000000000000
[ 5966.247170] RBP: 0000000000000000 R08: 0000000000000001 R09: 0000000000000000
[ 5966.247181] R10: ffff9077453d3a00 R11: 0000000000000001 R12: ffff9d82c1d77f58
[ 5966.247192] R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
[ 5966.247203] FS:  00007fa8147e9540(0000) GS:ffff9077dbcc0000(0000) knlGS:0000000000000000
[ 5966.247215] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5966.247224] CR2: 0000000000000000 CR3: 000000010633c000 CR4: 00000000000506e0
[ 5966.247236] Call Trace:
[ 5966.247243]  <TASK>
[ 5966.247249]  spa_fini+0x41/0x220 [zfs]
[ 5966.247327]  zfs_kmod_fini+0x67/0xc0 [zfs]
[ 5966.247382]  openzfs_fini+0xa/0xf14 [zfs]
[ 5966.247440]  __do_sys_delete_module+0x18f/0x298
[ 5966.247450]  ? exit_to_user_mode_prepare+0x32/0x178
[ 5966.247460]  do_syscall_64+0x3b/0xb8
[ 5966.247468]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[ 5966.247478] RIP: 0033:0x7fa8149107d7
[ 5966.247484] Code: 73 01 c3 48 8b 0d b9 f6 0b 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 b8 b0 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 89 f6 0b 00 f7 d8 64 89 01 48
[ 5966.247749] RSP: 002b:00007ffc9956c2a8 EFLAGS: 00000206 ORIG_RAX: 00000000000000b0
[ 5966.247991] RAX: ffffffffffffffda RBX: 000056266c2d2760 RCX: 00007fa8149107d7
[ 5966.248232] RDX: 000000000000000a RSI: 0000000000000800 RDI: 000056266c2d27c8
[ 5966.248487] RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000000
[ 5966.248715] R10: 00007fa814983ac0 R11: 0000000000000206 R12: 00007ffc9956c4d0
[ 5966.248942] R13: 00007ffc9956c8fb R14: 000056266c2d22a0 R15: 000056266c2d2760
[ 5966.249171]  </TASK>
[ 5966.249413] Modules linked in: zfs(POE-) zunicode(POE) zzstd(OE) zlua(OE) zavl(POE) icp(POE) zcommon(POE) znvpair(POE) spl(OE) rpcsec_gss_krb5(E) nfsv4(E) dns_resolver(E) nfs(E) fscache(E) netfs(E) socwatch2_15(OE) sep5(OE) socperf3(OE) intel_rapl_msr(E) intel_rapl_common(E) rfkill(E) pax(OE) ghash_clmulni_intel(E) aesni_intel(E) libaes(E) crypto_simd(E) cryptd(E) snd_pcm(E) snd_timer(E) joydev(E) snd(E) soundcore(E) serio_raw(E) pcspkr(E) sg(E) vboxguest(E) ac(E) evdev(E) binfmt_misc(E) nfsd(E) auth_rpcgss(E) nfs_acl(E) lockd(E) grace(E) msr(E) sunrpc(E) fuse(E) configfs(E) ip_tables(E) x_tables(E) autofs4(E) ext4(E) crc16(E) mbcache(E) jbd2(E) raid10(E) raid456(E) async_raid6_recov(E) async_memcpy(E) async_pq(E) async_xor(E) async_tx(E) xor(E) hid_generic(E) usbhid(E) hid(E) raid6_pq(E) libcrc32c(E) crc32c_generic(E) raid1(E) raid0(E) multipath(E) linear(E) md_mod(E) sd_mod(E) sr_mod(E) t10_pi(E) crc_t10dif(E) cdrom(E) crct10dif_generic(E) ata_generic(E) ohci_pci(E) virtio_net(E)
[ 5966.249446]  net_failover(E) vmwgfx(E) failover(E) ttm(E) ahci(E) ata_piix(E) libahci(E) ohci_hcd(E) ehci_pci(E) libata(E) ehci_hcd(E) virtio_pci(E) crct10dif_pclmul(E) crct10dif_common(E) crc32_pclmul(E) drm_kms_helper(E) crc32c_intel(E) usbcore(E) virtio_pci_modern_dev(E) cec(E) psmouse(E) scsi_mod(E) usb_common(E) i2c_piix4(E) virtio(E) drm(E) virtio_ring(E) scsi_common(E) battery(E) video(E) button(E)
[ 5966.253295] CR2: 0000000000000000

From the large end:

[  111.342398] VERIFY(zio_buf_cache[c] != NULL) failed
[  111.342413] PANIC at zio.c:234:zio_init()
[  111.342421] Showing stack for process 1522
[  111.342422] CPU: 5 PID: 1522 Comm: modprobe Kdump: loaded Tainted: P           OE     5.15.11badidea1 #1
[  111.342424] Hardware name: innotek GmbH VirtualBox/VirtualBox, BIOS VirtualBox 12/01/2006
[  111.342425] Call Trace:
[  111.342426]  <TASK>
[  111.342428]  dump_stack_lvl+0x46/0x5a
[  111.342433]  spl_panic+0xd1/0xe9 [spl]
[  111.342439]  ? pcpu_free_area+0x1f9/0x390
[  111.342442]  ? spl_kmem_cache_create+0x52b/0x6b0 [spl]
[  111.342445]  ? kfree+0x209/0x260
[  111.342448]  zio_init+0x2b3/0x2c0 [zfs]
[  111.342520]  ? 0xffffffffc1857000
[  111.342521]  spa_init+0x127/0x1d0 [zfs]
[  111.342582]  zfs_kmod_init+0x21/0x1170 [zfs]
[  111.342640]  openzfs_init+0xc/0x1000 [zfs]
[  111.342684]  do_one_initcall+0x44/0x1d0
[  111.342687]  ? __cond_resched+0x16/0x40
[  111.342689]  ? kmem_cache_alloc_trace+0x2cb/0x3d8
[  111.342691]  do_init_module+0x5c/0x268
[  111.342694]  __do_sys_finit_module+0xae/0x110
[  111.342695]  do_syscall_64+0x3b/0xb8
[  111.342698]  entry_SYSCALL_64_after_hwframe+0x44/0xae
[  111.342699] RIP: 0033:0x7ff8f456d9b9
[  111.342701] Code: 00 c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d a7 54 0c 00 f7 d8 64 89 01 48
[  111.342702] RSP: 002b:00007fffae9fa368 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[  111.342704] RAX: ffffffffffffffda RBX: 0000556c83cfcd30 RCX: 00007ff8f456d9b9
[  111.342705] RDX: 0000000000000000 RSI: 0000556c82557260 RDI: 000000000000000a
[  111.342705] RBP: 0000000000040000 R08: 0000000000000000 R09: 0000556c83cfce90
[  111.342706] R10: 000000000000000a R11: 0000000000000246 R12: 0000556c82557260
[  111.342706] R13: 0000000000000000 R14: 0000556c83cfcc00 R15: 0000556c83cfcd30
[  111.342707]  </TASK>
stale[bot] commented 1 year ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.