Many transient z_wr_int threads (~5 per second)

[colmbuckley]

System information

Type	Version/Name
Distribution Name	Debian
Distribution Version	Buster + Buster-Backports
Linux Kernel	5.3.0-0.bpo.2-amd64
Architecture	x86_64
ZFS Version	0.8.2 (debian version: 0.8.2-3~bpo10+1)
SPL Version	0.8.2 (debian version: 0.8.2-3~bpo10+1)

Describe the problem you're observing

Many transient z_wr_int threads seem to be being created and destroyed very rapidly; about 4 or 5 per second. First noticed that system load seemed a bit high (~6% on a Celeron G4920) when system is otherwise basically idle, then noticed that the active last PID was increasing fairly rapidly. Was tricky to identify that these were ZFS kernel threads as they were very short-lived; running a lot of "ps auxwww" in quick succession was able to catch them.

This is a Debian system, laid out basically as described at https://github.com/zfsonlinux/zfs/wiki/Debian-Buster-Root-on-ZFS.

Two pools - bpool (for GRUB /boot) and rpool (all other filesystems) on a single Kingston SUV500120G SSD. Autotrim is enabled on both pools.

I don't see similar behavior on other systems with very similar configurations. This system is running a Prometheus collector which performs many small writes to its database; the rate of thread creation does seem to decrease when this is disabled, so I guess it's related to disk write rate, but I don't think it's expected behavior?

Any additional configuration or logging I can supply to help track this down?

Describe how to reproduce the problem

Include any warning/errors/backtraces from the system logs

[colmbuckley]

Correction to kernel version: it's Debian's standard backports kernel - 5.3.9-2~bpo10+1 (2019-11-13)

[12101111]

Same on linux-5.4.5-gentoo + zfs 0.8.2. I can see them in htop

[colmbuckley]

(Of course, this might be WAI. Just seems new to me; I only have a few dozen ZoL systems.)

[behlendorf]

ZFS will create and destroy worker threads as they are needed, that behavior is expected. However, it sounds like your particular workload may be causing this to happen more frequently than usual. This shouldn't result in problems since the overhead involved in creating the threads is minimal. But if you would prefer the threads to be long livedyou can disable this behavior by setting the spl_taskq_thread_dynamic module option to 0.

[colmbuckley]

Aha, thanks @behlendorf

I'll play with that parameter to see how things look. Are there any expected performance or other benefits/losses from setting it?

[behlendorf]

Depending on your workload you might see a tiny performance improvement due to the threads being immediately available. However, for the vast majority of cases we haven't found that to be the case. The more noticeable impact will be an increased number of idle zfs related threads running on the system even when idle.

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[lnicola]

This is still creating about 10 threads per second on my otherwise idle system. I can try spl_taskq_thread_dynamic=0, but it seems like a waste of system resources to keep starting and killing threads like that.

[Shados]

I recently upgraded my kernel to 5.10.37 and zfs to 2.1.0-rc5, and found my PID numbers were rising into the hundreds of thousands after only being up for a couple of hours, where I can usually run my system for weeks and not get above 60k. I did some digging and found out that the source was transient z_wr_int and z_rd_int threads; despite no real change in workload.

I'm seeing ~30 such transient threads per second while idle, and potentially thousands if actually doing some IO-heavy work. Something is clearly very wonky here. I'm running with spl_taskq_thread_dynamic=0 for now to alleviate this.

[stale[bot]]

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

[lnicola]

Nothing changed here, bad bot.