faceless2
commented
2 years ago I am also seeing this.
I'd struggled with the issue from #9966 as well for months, even after setting these options
options zfs zfs_arc_min=17179869184
options zfs zfs_arc_max=17179869184
options zfs zfs_arc_meta_limit_percent=100
options zfs zfs_arc_dnode_limit_percent=75After upgrading the server (AMD chips, 96-cores, 128GB) from 20.04 to 22.04 (and thus ZFS to 2.1.4) and reverting all these options to their defaults, the issue of having 96 arc_prune processes running flat out is gone - I now have one arc_prune process running flat out. Don't get me wrong, that's a huge improvement - I can at least log in and get diagnostics. But it's not there yet.
Heres cat /proc/spl/kstat/zfs/arcstats | egrep '(dnode|arc_meta)' - very similar to the OP. In our case we're running a large regression test which creates tens of thousands of files over a few minutes. It doesn't happen every time despite almost identical workload.
dnode_size 4 6383114272
arc_meta_used 4 21562042592
arc_meta_limit 4 17179869184
arc_dnode_limit 4 12884901888
arc_meta_max 4 22070295584
arc_meta_min 4 16777216I've now tried reapplying the zfs_arc_min and zfs_arc_max options above and will keep an eye on it - next time I'll post arc_summary output before dropping caches, but if there's any other diagnostic info that would help please let me know.
shodanshok
coot
luismcv
unkownerror1
IZSkiSurfer
amotin
olidal
drescherjm
ipaqmaster
System information
Describe the problem you're observing
I've been using ZFS for the root fs of my desktop, with ARC limited to 2GB, for several months without any issues until now. When I run disk intensive tasks, like borg-backup, duc, find, etc... I can see arc_prune and arc_evict working intermittently, with peaks of about 15% of a thread each, and they manage to keep the arc within limits:
And once the tasks are finished they both go to sleep. All normal so far.
But yesterday I decided to try another backup tool, Kopia, and I know what it does while doing a backup that makes ARC going out of control. Both arc_prune and arc_evict start using 100% of a CPU thread each. And despite that, they don't manage to keep the memory limits within range, only reaching some kind of balance at around 3.6GB.
But even after Kopia has finished or I have aborted it, the problem keeps going on indefinitely, even though there's no processes disk activity anymore (iostat and my system's disk led both show some continuous activity though, so it seems it's not just the CPU what they're using).
NOTES:
echo 3 > /proc/sys/vm/drop_cachesstops it, until I run a backup again.Describe how to reproduce the problem
Run a Kopia backup of a ZFS filesytem with many files and a low(ish) memory limit for ARC.