search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.42k stars 1.73k forks source link

Kernel panic on module load: ZFS 2.1.11 on RHEL8.7 4.18.0-425.13.1 #14812

Open draeath opened 1 year ago

draeath commented 1 year ago

System information

Type Version/Name
Distribution Name RHEL
Distribution Version 8.7
Kernel Version 4.18.0-425.13.1.el8_7
Architecture x86_64
OpenZFS Version 2.1.11

Describe the problem you're observing

Kernel panic on module load (during boot) or when doing zfs import if system was booted without ZFS installed.

BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
  ... elided output, full output later
Kernel panic - not syncing: Fatal exception in interrupt

Describe how to reproduce the problem

  1. Install zfs 2.1.11 via DKMS
  2. Allow automatic module loading, or manually load the module
  3. Panic occurs

I already had ZFS via DKMS before this occurred, after updating packages I noticed that 'dkms status' had some odd output (did not save it). To troubleshoot I booted to emergency.target, disabled the dependent services, and completely uninstalled ZFS. Boot was successful, I reinstalled zfs-dkms, zfs-dracut, and zfs packages. I did a dkms autoinstall -m zfs -v 2.1.11 -k <varies> for each installed kernel. Finally, I did dracut --regenerate-all --force. I did NOT yet reboot, but ran zfs import to ensure the module worked. At this point, a panic occurs. Subsequent boots see the same panic, as expected.

Include any warning/errors/backtraces from the system logs

Output is from RS232 console, line "Modules linked in:" is truncated

[ 1926.930705] PGD 0 P4D 0 
[ 1926.933540] Oops: 0002 [#1] SMP PTI
[ 1926.937441] CPU: 16 PID: 160217 Comm: z_null_int Tainted: P           OE    --------- -  - 4.18.0-425.13.1.el8_7.x86_64 #1
[ 1926.949798] Hardware name: Dell Inc. PowerEdge R720xd/0VWT90, BIOS 2.9.0 12/06/2019
[ 1926.958355] RIP: 0010:_raw_spin_lock_irqsave+0x1e/0x40
[ 1926.964099] Code: cf 80 0b 08 eb 88 90 90 90 90 90 90 66 66 66 66 90 53 9c 58 66 66 90 66 90 48 89 c3 fa 66 66 90 66 66 90 31 c0 ba 01 00 00 00 <f0> 0f b1 17 75 09 48 89 d8 5b e9 e3 02 21 00 89 c6 e8 7c 4b 76 ff
[ 1926.985102] RSP: 0018:ffffa1dd067a4d60 EFLAGS: 00010046
[ 1926.990955] RAX: 0000000000000000 RBX: 0000000000000282 RCX: 0000000000000000
[ 1926.998935] RDX: 0000000000000001 RSI: ffffffffc0d44050 RDI: 0000000000000000
[ 1927.006918] RBP: ffff94c21c494558 R08: ffff94c21c494558 R09: 0000000000000000
[ 1927.014898] R10: 0000000000000002 R11: 0000000000000000 R12: ffff94c21c494100
[ 1927.022880] R13: ffff94c21c494100 R14: ffffffffc0d44050 R15: 0000000000000000
[ 1927.030863] FS:  0000000000000000(0000) GS:ffff94d0bfa00000(0000) knlGS:0000000000000000
[ 1927.039914] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1927.046339] CR2: 0000000000000000 CR3: 00000003d5610002 CR4: 00000000000606e0
[ 1927.054322] Call Trace:
[ 1927.057056]  <IRQ>
[ 1927.059313]  taskq_dispatch_ent+0x27/0x1c0 [spl]
[ 1927.064487]  ? recalibrate_cpu_khz+0x10/0x10
[ 1927.069265]  ? zio_taskq_member.isra.11.constprop.17+0x60/0x60 [zfs]
[ 1927.076477]  spa_taskq_dispatch_ent+0x64/0xb0 [zfs]
[ 1927.082015]  zio_taskq_dispatch+0x61/0xa0 [zfs]
[ 1927.087168]  zio_delay_interrupt+0x44/0x190 [zfs]
[ 1927.092507]  ? vdev_disk_dio_put+0x2d/0x60 [zfs]
[ 1927.097755]  ? kfree+0xd3/0x250
[ 1927.101270]  vdev_disk_dio_put+0x46/0x60 [zfs]
[ 1927.106324]  blk_update_request+0x224/0x380
[ 1927.111012]  scsi_end_request+0x28/0x110
[ 1927.115405]  scsi_io_completion+0x7e/0x590
[ 1927.119987]  blk_complete_reqs+0x35/0x50
[ 1927.124375]  __do_softirq+0xdc/0x2cf
[ 1927.128375]  irq_exit_rcu+0xd5/0xe0
[ 1927.132283]  irq_exit+0xa/0x10
[ 1927.135696]  do_IRQ+0x7f/0xd0
[ 1927.139014]  common_interrupt+0xf/0xf
[ 1927.143102]  </IRQ>
[ 1927.145448] RIP: 0010:SHA256TransformBlocks+0x1144/0x1310 [icp]
[ 1927.152080] Code: 89 d7 44 03 64 bd 00 44 31 e9 41 c1 cd 09 45 09 ce 44 31 e9 45 21 cf 45 01 e2 45 21 c6 44 01 e1 45 09 fe 48 8d 7f 01 44 01 f1 <44> 8b 6c 24 3c 44 8b 64 24 30 45 89 ef 41 c1 ed 03 41 c1 cf 07 45
[ 1927.173082] RSP: 0018:ffffa1dd0e6a3900 EFLAGS: 00000206 ORIG_RAX: ffffffffffffffdd
[ 1927.181549] RAX: 0000000076983b79 RBX: 000000004677e42a RCX: 0000000060df237e
[ 1927.189529] RDX: 000000005b97523d RSI: ffff94c1f88d13c0 RDI: 000000000000001e
[ 1927.197511] RBP: ffffffffc07e0d00 R08: 000000002db58534 R09: 00000000c358ead6
[ 1927.205494] R10: 000000005902de69 R11: 000000004fc0da51 R12: 000000007b0163ef
[ 1927.213477] R13: 000000005d48f56e R14: 000000004b95c234 R15: 0000000043104214
[ 1927.221460]  ? SHA2Update+0x1c4/0x1e0 [icp]
[ 1927.226137]  ? sha_incremental+0x16/0x20 [zfs]
[ 1927.231185]  ? abd_iterate_func.part.13+0xc1/0x190 [zfs]
[ 1927.237177]  ? sa_register_update_callback_locked+0x20/0x20 [zfs]
[ 1927.244072]  ? abd_checksum_SHA256+0x51/0xb0 [zfs]
[ 1927.249508]  ? __blk_mq_run_hw_queue+0x2b/0x70
[ 1927.254470]  ? __blk_mq_delay_run_hw_queue+0x141/0x160
[ 1927.260217]  ? abd_copy_from_buf_off_cb+0x18/0x20 [zfs]
[ 1927.266113]  ? zio_checksum_error_impl+0x28f/0x650 [zfs]
[ 1927.272140]  ? sha_incremental+0x20/0x20 [zfs]
[ 1927.277188]  ? vdev_disk_io_start+0x3d6/0x8f0 [zfs]
[ 1927.283040]  ? recalibrate_cpu_khz+0x10/0x10
[ 1927.288091]  ? zio_vdev_io_start+0xf5/0x300 [zfs]
[ 1927.293710]  ? vdev_queue_io_to_issue+0x3ba/0xb50 [zfs]
[ 1927.299915]  ? zio_checksum_error+0x60/0xc0 [zfs]
[ 1927.305546]  ? zio_checksum_verify+0x36/0x140 [zfs]
[ 1927.311356]  ? _cond_resched+0x15/0x30
[ 1927.315815]  ? mutex_lock+0xe/0x30
[ 1927.319885]  ? zio_wait_for_children+0x87/0xc0 [zfs]
[ 1927.325786]  ? zio_vdev_io_assess+0x20/0x250 [zfs]
[ 1927.331489]  ? zio_execute+0x90/0xf0 [zfs]
[ 1927.336408]  ? taskq_thread+0x2dd/0x510 [spl]
[ 1927.341537]  ? wake_up_q+0x70/0x70
[ 1927.345587]  ? zio_taskq_member.isra.11.constprop.17+0x60/0x60 [zfs]
[ 1927.353012]  ? taskq_thread_spawn+0x50/0x50 [spl]
[ 1927.358514]  ? kthread+0x10b/0x130
[ 1927.362555]  ? set_kthread_struct+0x50/0x50
[ 1927.367467]  ? ret_from_fork+0x35/0x40
[ 1927.371893] Modules linked in: team_mode_loadbalance team nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nf_tables_set nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 ne
[ 1927.469157] CR2: 0000000000000000
[ 1927.473151] ---[ end trace 43d332c638f86fe5 ]---
[ 1927.483756] RIP: 0010:_raw_spin_lock_irqsave+0x1e/0x40
[ 1927.489779] Code: cf 80 0b 08 eb 88 90 90 90 90 90 90 66 66 66 66 90 53 9c 58 66 66 90 66 90 48 89 c3 fa 66 66 90 66 66 90 31 c0 ba 01 00 00 00 <f0> 0f b1 17 75 09 48 89 d8 5b e9 e3 02 21 00 89 c6 e8 7c 4b 76 ff
[ 1927.511342] RSP: 0018:ffffa1dd067a4d60 EFLAGS: 00010046
[ 1927.517469] RAX: 0000000000000000 RBX: 0000000000000282 RCX: 0000000000000000
[ 1927.525735] RDX: 0000000000000001 RSI: ffffffffc0d44050 RDI: 0000000000000000
[ 1927.533994] RBP: ffff94c21c494558 R08: ffff94c21c494558 R09: 0000000000000000
[ 1927.542264] R10: 0000000000000002 R11: 0000000000000000 R12: ffff94c21c494100
[ 1927.550528] R13: ffff94c21c494100 R14: ffffffffc0d44050 R15: 0000000000000000
[ 1927.558795] FS:  0000000000000000(0000) GS:ffff94d0bfa00000(0000) knlGS:0000000000000000
[ 1927.568139] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 1927.574847] CR2: 0000000000000000 CR3: 00000003d5610002 CR4: 00000000000606e0
[ 1927.583127] Kernel panic - not syncing: Fatal exception in interrupt
[ 1927.847990] Kernel Offset: 0x1c600000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 1927.865515] ---[ end Kernel panic - not syncing: Fatal exception in interrupt ]---
draeath commented 1 year ago

Note, 4.18.0-372.26.1.el8_6 kernel boots fine with this module version.

rincebrain commented 1 year ago

I would guess this is another kABI issue, but don't have a RHEL8 system readily available to check. Try making DKMS delete and rebuild if you're using DKMS; and if not, try using DKMS? (I know you said you told it force, but I've seen DKMS decide the modules there were fine and it didn't need to rebuild before even when I asked it to, so please try removing them outright, being sure they're gone from every kernel, then installing again.)

draeath commented 1 year ago

I am using DKMS, and I did that already :P

I cleaned up the modules manually after uninstall, just didn't include that detail - sorry.

tonyhutter commented 1 year ago

@draeath you need to update your kernel. You're running:

4.18.0-425.13.1

The kmods are built for 4.18.0-425.19.2

rincebrain commented 1 year ago

If these are built with DKMS, and it's indeed loading the DKMS versions, then it shouldn't matter how old it is as long as they're built against the right version, no?

(I suppose the question also becomes, do the prebuilt kmods break the same way if you swap to those?)

draeath commented 1 year ago

Indeed, I'm avoiding the prebuilt kABI-tracking modules intentionally, trying to avoid this very sort of problem.

(If I did use the prebuilt, would the module loader even find them?)

I'm hesitant to try as this came up on our log aggregation host - something I would like to avoid downtime on.

(complicating factor: we use a satellite instance that tends to be a month or two behind the upstream cdn and I do not have access to poke it for fresh kernels)

tonyhutter commented 1 year ago

The kmods are built for 4.18.0-425.19.2

@draeath sorry, I totally glossed over that you were using the dkms modules, not kmods. I tried installing the 2.1.11 DKMS modules on EL8 (Almalinux 8) with the 4.18.0-425.19.2.el8_7.x86_64 kernel, and I was able to import a pool without issue.

rincebrain commented 1 year ago

Nothing up my sleeve...

[root@rhel8 ~]# dkms status
zfs/2.1.11, 4.18.0-425.13.1.el8_7.x86_64, x86_64: installed
[root@rhel8 ~]# uname -a
Linux rhel8 4.18.0-425.13.1.el8_7.x86_64 #1 SMP Thu Feb 2 13:01:45 EST 2023 x86_64 x86_64 x86_64 GNU/Linux
[root@rhel8 ~]# zpool status
  pool: mypool
 state: ONLINE
config:

        NAME           STATE     READ WRITE CKSUM
        mypool         ONLINE       0     0     0
          /root/bees1  ONLINE       0     0     0

errors: No known data errors
[root@rhel8 ~]# zpool version
zfs-2.1.11-1
zfs-kmod-2.1.11-1

Just using the dkms packages from the repo.

Could you possibly share, in full, the output of rpm -qa | egrep '(kernel|zfs|zpool)' | sort and dkms status?

(For reference, on my system newly-installed and then manually rebooted and installed the older kernel, I get:

abrt-addon-kerneloops-2.10.9-21.el8.x86_64
kernel-4.18.0-425.13.1.el8_7.x86_64
kernel-4.18.0-425.19.2.el8_7.x86_64
kernel-core-4.18.0-425.13.1.el8_7.x86_64
kernel-core-4.18.0-425.19.2.el8_7.x86_64
kernel-devel-4.18.0-425.13.1.el8_7.x86_64
kernel-devel-4.18.0-425.19.2.el8_7.x86_64
kernel-headers-4.18.0-425.19.2.el8_7.x86_64
kernel-modules-4.18.0-425.13.1.el8_7.x86_64
kernel-modules-4.18.0-425.19.2.el8_7.x86_64
kernel-tools-4.18.0-425.19.2.el8_7.x86_64
kernel-tools-libs-4.18.0-425.19.2.el8_7.x86_64
libzfs5-2.1.11-1.el8.x86_64
libzpool5-2.1.11-1.el8.x86_64
zfs-2.1.11-1.el8.x86_64
zfs-dkms-2.1.11-1.el8.noarch
zfs-release-2-2.el8.noarch

)

e: that's interesting, I explicitly told it to install kernel, -devel, and -headers of 13.1, and it only did the first two. Time to try rebuilding with the older headers in place...)

e2: nope, doesn't change, it doesn't panic for me.

draeath commented 1 year ago 
# dkms status
zfs/2.1.11, 4.18.0-372.26.1.el8_6.x86_64, x86_64: installed
zfs/2.1.11, 4.18.0-372.9.1.el8.x86_64, x86_64: installed
zfs/2.1.11, 4.18.0-425.13.1.el8_7.x86_64, x86_64: installed
# rpm -qa | egrep '(kernel|zfs|zpool)' | sort
kernel-4.18.0-372.26.1.el8_6.x86_64
kernel-4.18.0-372.9.1.el8.x86_64
kernel-4.18.0-425.13.1.el8_7.x86_64
kernel-core-4.18.0-372.26.1.el8_6.x86_64
kernel-core-4.18.0-372.9.1.el8.x86_64
kernel-core-4.18.0-425.13.1.el8_7.x86_64
kernel-devel-4.18.0-372.26.1.el8_6.x86_64
kernel-devel-4.18.0-372.9.1.el8.x86_64
kernel-devel-4.18.0-425.13.1.el8_7.x86_64
kernel-headers-4.18.0-425.13.1.el8_7.x86_64
kernel-modules-4.18.0-372.26.1.el8_6.x86_64
kernel-modules-4.18.0-372.9.1.el8.x86_64
kernel-modules-4.18.0-425.13.1.el8_7.x86_64
kernel-tools-4.18.0-425.13.1.el8_7.x86_64
kernel-tools-libs-4.18.0-425.13.1.el8_7.x86_64
libzfs5-2.1.11-1.el8.x86_64
libzpool5-2.1.11-1.el8.x86_64
zfs-2.1.11-1.el8.x86_64
zfs-dkms-2.1.11-1.el8.noarch
zfs-dracut-2.1.11-1.el8.noarch
zfs-release-el-2-1.noarch

Here's that information. Also, just to make sure, this is the only enabled ZFS repo:

[zfs]
name=ZFS on Linux for EL$releasever - dkms
baseurl=http://download.zfsonlinux.org/epel/$releasever/$basearch/
enabled=1
metadata_expire=7d
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-zfsonlinux
rincebrain commented 1 year ago

The next one, I suppose, becomes "if you do find /lib/modules/ -iname zfs.ko.*, what does it say?"

In particular, I'm wondering if it's "helpfully" linking the module versions from the older kernel in the newer one's directory such that they're earlier in the module search path.

draeath commented 1 year ago

Oh, weak-updates? I could see that.

I'll have a look. I could also outright disable them for these module in the DKMS configuration, I have done that for nvidia modukes in the past

draeath commented 1 year ago

The next one, I suppose, becomes "if you do find /lib/modules/ -iname zfs.ko.*, what does it say?"

In particular, I'm wondering if it's "helpfully" linking the module versions from the older kernel in the newer one's directory such that they're earlier in the module search path.

# for i in $(find /lib/modules/ -iname zfs.ko.*); do stat "$i" | grep 'File: '; done
  File: /lib/modules/4.18.0-372.9.1.el8.x86_64/extra/zfs.ko.xz
  File: /lib/modules/4.18.0-372.26.1.el8_6.x86_64/extra/zfs.ko.xz
  File: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/weak-updates/zfs.ko.xz -> /lib/modules/4.18.0-372.26.1.el8_6.x86_64/extra/zfs.ko.xz
  File: /lib/modules/4.18.0-425.13.1.el8_7.x86_64/extra/zfs.ko.xz

That's odd. I must have missed something while cleaning up - you shouldn't have the module in both places, right?

(I've run into problems with weak-updates before. anyone know why the dkms configuration you ship in the RPMs doesn't disable them for the zfs modules? googling around shows me someone did so on their fork, or at least tried?)

EDIT:

# grep -i weak /usr/src/zfs-2.1.11/dkms.conf
NO_WEAK_MODULES="yes"

Now I'm thoroughly confused how those got there. I removed them manually, uninstalled the module for that kernel via dkms, reinstalled... and it looks right now - real files under extra instead of symlinks under weak-updates

I'll try to see if that resolves the kernel panic.

draeath commented 1 year ago

Bingo, that was the problem. I don't understand how those weak-update symlinks were present given the zfs-dkms configuration turns them off (and doing a dkms uninstall, removing them, and then doing a dkms autoinstall does not recreate them).

rincebrain commented 1 year ago

Maybe we need a different way of setting NO_WEAK_MODULES? Clearly it didn't listen.

draeath commented 1 year ago

While I think it's unrelated, it's not impossible something else is up with this host that caused this behavior: we found today that there was a busted weak-updates symlink for a veeam module as well on this same host.