behlendorf
commented
11 years ago @akorn Could you try resizing spl_kmem_cache taskq, this was one significant changed between 0.6.0.90 and 0.6.0.91. Basically, these allocations used to be handled by the kworker threads (1 per cpu) but were moved to a single taskq (1 thread by default). So it's possible this might be a bottleneck on a larger or busy system.
You should be able to make this change to the DKMS sources on your Ubuntu system and just have DKMS rebuild everything. It increases the numbers of threads to 1 per cpu and effectively removes the queue depth limit form the taskq.
diff --git a/module/spl/spl-kmem.c b/module/spl/spl-kmem.c
index cc5961e..1f8839f 100644
--- a/module/spl/spl-kmem.c
+++ b/module/spl/spl-kmem.c
@@ -2409,7 +2409,8 @@ spl_kmem_init(void)
init_rwsem(&spl_kmem_cache_sem);
INIT_LIST_HEAD(&spl_kmem_cache_list);
spl_kmem_cache_taskq = taskq_create("spl_kmem_cache",
- 1, maxclsyspri, 1, 32, TASKQ_PREPOPULATE);
+ 100, maxclsyspri, 1, INT_MAX,
+ TASKQ_PREPOPULATE | TASKQ_THREADS_CPU_PCT);
spl_register_shrinker(&spl_kmem_cache_shrinker);
akorn
Hi,
since I upgraded to 0.6.0.92 from 0.6.0.91, I see this in top:
(I have four cores and one of them is continually busy with spl_kmem_cache.)
Additionally, some operations take ridiculously long (if they finish at all).
I have some sysrq-w output: