NFS share of ZFS hangs randomly under load

laf070810 commented 2 months ago

System information

Type	Version/Name
Distribution Name	AlmaLinux
Distribution Version	9.4
Kernel Version	Linux 5.14.0-427.26.1.el9_4.x86_64
Architecture	x86_64
OpenZFS Version	zfs-kmod-2.1.15-3

Describe the problem you're observing

Server A is the NFS server sharing a ZFS pool. Server B, C, D, ... are NFS clients mounting the share from server A. If the NFS clients have some loads accessing the NFS share, the updatedb and some or all of the nfsd processes on the NFS server may hang in D state forever and some or all of the clients may hang forever on any command involving the NFS share, even when the NFS mount is a soft mount rather than a hard mount. I guess the cause might be some deadlock in ZFS like #11003 .

Describe how to reproduce the problem

No exact condition that can reproduce the problem is found. But high IO load of the NFS is suspected. And this issue has happened several times in our cases.

Include any warning/errors/backtraces from the system logs

The updatedb and some nfsd processes are hung in D state:

➜  ~ ps aux | grep nfsd
root       13785  0.0  0.0   5456  3072 ?        Ss   Aug14   0:00 /usr/sbin/nfsdcld
root       87360  0.0  0.0      0     0 ?        S    Aug14   4:30 [nfsd]
root       87363  0.0  0.0      0     0 ?        S    Aug14   6:42 [nfsd]
root       87365  0.0  0.0      0     0 ?        S    Aug14  11:51 [nfsd]
root       87367  0.1  0.0      0     0 ?        S    Aug14  23:12 [nfsd]
root       87369  0.2  0.0      0     0 ?        S    Aug14  46:40 [nfsd]
root       87370  0.6  0.0      0     0 ?        D    Aug14 130:58 [nfsd]
root       87372  0.5  0.0      0     0 ?        D    Aug14  99:36 [nfsd]
root       87374  0.0  0.0      0     0 ?        D    Aug14   8:32 [nfsd]
root      412278  3.1  0.0 222436  1536 ?        DN   Aug26   3:18 /usr/bin/updatedb -f sysfs tmpfs bdev proc cgroup cgroup2 cpuset devtmpfs configfs debugfs tracefs securityfs sockfs bpf pipefs ramfs hugetlbfs devpts autofs efivarfs mqueue selinuxfs resctrl pstore fuse fusectl rpc_pipefs nfsd binfmt_misc overlay afs
root     1870894  0.0  0.0      0     0 ?        D    Aug25   0:00 [kworker/u388:4+nfsd4]

The stack trace of the hung processes:

➜  ~ sudo cat /proc/412278/stack
[<0>] __cv_timedwait_common+0x12d/0x170 [spl]
[<0>] __cv_timedwait_io+0x15/0x20 [spl]
[<0>] zio_wait+0x130/0x290 [zfs]
[<0>] dmu_buf_hold+0x5f/0x90 [zfs]
[<0>] zap_lockdir+0x4e/0xc0 [zfs]
[<0>] zap_lookup+0x4a/0x100 [zfs]
[<0>] zfs_dirent_lock+0x553/0x6b0 [zfs]
[<0>] zfs_dirlook+0x88/0x2c0 [zfs]
[<0>] zfs_lookup+0x24a/0x400 [zfs]
[<0>] zpl_lookup+0xc9/0x1f0 [zfs]
[<0>] __lookup_slow+0x81/0x130
[<0>] walk_component+0x158/0x1d0
[<0>] path_lookupat+0x6e/0x1c0
[<0>] filename_lookup+0xcf/0x1d0
[<0>] vfs_statx+0x8d/0x170
[<0>] vfs_fstatat+0x54/0x70
[<0>] __do_sys_newfstatat+0x26/0x60
[<0>] do_syscall_64+0x59/0x90
[<0>] entry_SYSCALL_64_after_hwframe+0x72/0xdc

➜  ~ sudo cat /proc/1870894/stack
[<0>] nfsd4_shutdown_callback+0xa3/0x120 [nfsd]
[<0>] __destroy_client+0x1f3/0x290 [nfsd]
[<0>] nfs4_process_client_reaplist+0xa1/0x110 [nfsd]
[<0>] nfsd4_state_shrinker_worker+0xee/0x120 [nfsd]
[<0>] process_one_work+0x1e2/0x3b0
[<0>] worker_thread+0x50/0x3a0
[<0>] kthread+0xdd/0x100
[<0>] ret_from_fork+0x29/0x50

laf070810 commented 1 week ago

Closing this issue as it seems to be a device-specific problem. The issue disappeared after banning a specific NFS client, though we have no idea about what's wrong with that machine.

amotin commented 1 week ago

[<0>] __cv_timedwait_common+0x12d/0x170 [spl]
[<0>] __cv_timedwait_io+0x15/0x20 [spl]
[<0>] zio_wait+0x130/0x290 [zfs]
[<0>] dmu_buf_hold+0x5f/0x90 [zfs]
[<0>] zap_lockdir+0x4e/0xc0 [zfs]

Looks like it is waiting for some disk read, but can't say why it stuck somewhere. When you was saying "device-specific", my first though was about your storage device, not client.

openzfs / zfs