Closed JKDingwall closed 1 month ago
I don't see how can it be related to lz4 or CPUs getting stuck, but since you mentioned heavy ballooning I have to mention that ZFS is currently quite reluctant to react on kernel's memory pressure. See zfs_arc_shrinker_limit
and https://github.com/openzfs/zfs/pull/16197 . I wonder if system is over-provisioned that much, can it cause some swapping on a VM host side, that would look like random CPU freezes for the guest?
With some further investigation it seems trying to do anything 'clever' with the zfsarc{min,max} tunables results in an unhappy system. I'd guess that the common error in LZ4_uncompress_unknownOutputSize could be something to do with the result of a memory allocation not being checked for success but I've not looked in to that further. Leaving those tunables alone and the behaviour seems stable as the dom0 memory balloons. I've patched our zfs build to discourage tinkering:
diff --git a/module/zfs/arc.c b/module/zfs/arc.c
index 195364013..a7fb0c449 100644
--- a/module/zfs/arc.c
+++ b/module/zfs/arc.c
@@ -10662,10 +10662,10 @@ EXPORT_SYMBOL(arc_add_prune_callback);
EXPORT_SYMBOL(arc_remove_prune_callback);
ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, min, param_set_arc_min,
- spl_param_get_u64, ZMOD_RW, "Minimum ARC size in bytes");
+ spl_param_get_u64, ZMOD_RD, "Minimum ARC size in bytes");
ZFS_MODULE_PARAM_CALL(zfs_arc, zfs_arc_, max, param_set_arc_max,
- spl_param_get_u64, ZMOD_RW, "Maximum ARC size in bytes");
+ spl_param_get_u64, ZMOD_RD, "Maximum ARC size in bytes");
ZFS_MODULE_PARAM(zfs_arc, zfs_arc_, meta_balance, UINT, ZMOD_RW,
"Balance between metadata and data on ghost hits.");
For the time being I'd say this is a 'user error' problem and the behaviour with defaults is good.
System information
Describe the problem you're observing
The system is virtual machine host with block devices allocated as zvols, during guest startup several instances of the following trace are logged and I/O becomes unresponsive. There are also some earlier reports of hung tasks but I don't know if these are related. The configuration is a new build of our existing environment with both zfs (was 2.1.15) and the kernel (was 5.15.0) being updated. I did see https://github.com/openzfs/zfs/issues/12775 but that thread has been quiet for 18+ months.
Describe how to reproduce the problem
Provision zvols for virtual machine (Xen) block storage and then try and boot the guests. For legacy reasons the zpool is initially created with zfs 0.8.3(?) and then
zpool upgrade
d during the first boot. All zpool features are enabled. We are using a local build of the kernel in order to patch in AUFS support. What is also happening during the vm start is that the dom0 memory is ballooning down and it seems that the max_arc_size (un)tunable can end up being = system RAM. Is that a problem?16 * 1024^3 = 17179869184
Include any warning/errors/backtraces from the system logs
On the console:
Kernel log: