search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.42k stars 1.72k forks source link

zfs hang on low/medium load #9172

Closed rnz closed 1 year ago

rnz commented 5 years ago

System information

Type Version/Name
Distribution Name Debian
Distribution Version Debian GNU/Linux 9.6 (stretch)
Linux Kernel 4.15.18-9-pve #1 SMP PVE 4.15.18-30 (Thu, 15 Nov 2018 13:32:46 +0100) x86_64 GNU/Linux
Architecture x86_64
ZFS Version 0.7.12-1
SPL Version 0.7.12-1

Hardware

# lscpu 
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                128
On-line CPU(s) list:   0-127
Thread(s) per core:    2
Core(s) per socket:    32
Socket(s):             2
NUMA node(s):          8
Vendor ID:             AuthenticAMD
CPU family:            23
Model:                 1
Model name:            AMD EPYC 7601 32-Core Processor
Stepping:              2
CPU MHz:               2661.997
CPU max MHz:           2200.0000
CPU min MHz:           1200.0000
BogoMIPS:              4399.73
Virtualization:        AMD-V
L1d cache:             32K
L1i cache:             64K
L2 cache:              512K
L3 cache:              8192K
NUMA node0 CPU(s):     0-7,64-71
NUMA node1 CPU(s):     8-15,72-79
NUMA node2 CPU(s):     16-23,80-87
NUMA node3 CPU(s):     24-31,88-95
NUMA node4 CPU(s):     32-39,96-103
NUMA node5 CPU(s):     40-47,104-111
NUMA node6 CPU(s):     48-55,112-119
NUMA node7 CPU(s):     56-63,120-127
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca
# free -m
              total        used        free      shared  buff/cache   available
Mem:        2052077      176121     1848600         672       27355     1877601
Swap:             0           0           0

Describe the problem you're observing

The IO hangs random. ZFS Volume is inaccessible, but ZFS Filesystems is accessible

Describe how to reproduce the problem

1) Create two zfs volume 2) one zfs volume set compression=on(lz4) 3) generate havy load in both zvol, 4) Wait until zvol is hung

Include any warning/errors/backtraces from the system logs

[Ср авг 14 06:54:48 2019] INFO: task zvol:2328 blocked for more than 120 seconds.
--
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] zvol            D    0  2328      2 0x80000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  rwsem_down_read_failed+0x10a/0x170
[Ср авг 14 06:54:48 2019]  ? mutex_lock+0x12/0x40
[Ср авг 14 06:54:48 2019]  call_rwsem_down_read_failed+0x18/0x30
[Ср авг 14 06:54:48 2019]  ? call_rwsem_down_read_failed+0x18/0x30
[Ср авг 14 06:54:48 2019]  ? spl_kmem_cache_free+0x13c/0x1c0 [spl]
[Ср авг 14 06:54:48 2019]  down_read+0x20/0x40
[Ср авг 14 06:54:48 2019]  dbuf_read+0x2fe/0x930 [zfs]
[Ср авг 14 06:54:48 2019]  ? dmu_buf_hold_array_by_dnode+0x14b/0x470 [zfs]
[Ср авг 14 06:54:48 2019]  dmu_buf_will_dirty+0x111/0x130 [zfs]
[Ср авг 14 06:54:48 2019]  dmu_write_uio_dnode+0x6b/0x140 [zfs]
[Ср авг 14 06:54:48 2019]  zvol_write+0x1a7/0x640 [zfs]
[Ср авг 14 06:54:48 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Ср авг 14 06:54:48 2019]  ? wake_up_q+0x80/0x80
[Ср авг 14 06:54:48 2019]  kthread+0x105/0x140
[Ср авг 14 06:54:48 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ? do_syscall_64+0x73/0x130
[Ср авг 14 06:54:48 2019]  ? SyS_exit_group+0x14/0x20
[Ср авг 14 06:54:48 2019]  ret_from_fork+0x22/0x40
[Ср авг 14 06:54:48 2019] INFO: task z_wr_iss:2869 blocked for more than 120 seconds.
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] z_wr_iss        D    0  2869      2 0x80000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  ? avl_nearest+0x2b/0x30 [zavl]
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  rwsem_down_write_failed+0x208/0x390
[Ср авг 14 06:54:48 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Ср авг 14 06:54:48 2019]  call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Ср авг 14 06:54:48 2019]  down_write+0x2d/0x40
[Ср авг 14 06:54:48 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Ср авг 14 06:54:48 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Ср авг 14 06:54:48 2019]  ? mutex_lock+0x12/0x40
[Ср авг 14 06:54:48 2019]  zio_ready+0x49/0x430 [zfs]
[Ср авг 14 06:54:48 2019]  ? taskq_member+0x18/0x30 [spl]
[Ср авг 14 06:54:48 2019]  zio_execute+0x95/0xf0 [zfs]
[Ср авг 14 06:54:48 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Ср авг 14 06:54:48 2019]  ? wake_up_q+0x80/0x80
[Ср авг 14 06:54:48 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Ср авг 14 06:54:48 2019]  kthread+0x105/0x140
[Ср авг 14 06:54:48 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ret_from_fork+0x22/0x40
[Ср авг 14 06:54:48 2019] INFO: task z_wr_iss:2872 blocked for more than 120 seconds.
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] z_wr_iss        D    0  2872      2 0x80000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  ? avl_nearest+0x2b/0x30 [zavl]
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  rwsem_down_write_failed+0x208/0x390
[Ср авг 14 06:54:48 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Ср авг 14 06:54:48 2019]  call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  down_write+0x2d/0x40
[Ср авг 14 06:54:48 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Ср авг 14 06:54:48 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Ср авг 14 06:54:48 2019]  ? mutex_lock+0x12/0x40
[Ср авг 14 06:54:48 2019]  zio_ready+0x49/0x430 [zfs]
[Ср авг 14 06:54:48 2019]  ? taskq_member+0x18/0x30 [spl]
[Ср авг 14 06:54:48 2019]  zio_execute+0x95/0xf0 [zfs]
[Ср авг 14 06:54:48 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Ср авг 14 06:54:48 2019]  ? wake_up_q+0x80/0x80
[Ср авг 14 06:54:48 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Ср авг 14 06:54:48 2019]  kthread+0x105/0x140
[Ср авг 14 06:54:48 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ret_from_fork+0x22/0x40
[Ср авг 14 06:54:48 2019] INFO: task z_wr_iss:2882 blocked for more than 120 seconds.
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] z_wr_iss        D    0  2882      2 0x80000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  ? avl_nearest+0x2b/0x30 [zavl]
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  rwsem_down_write_failed+0x208/0x390
[Ср авг 14 06:54:48 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Ср авг 14 06:54:48 2019]  call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Ср авг 14 06:54:48 2019]  down_write+0x2d/0x40
[Ср авг 14 06:54:48 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Ср авг 14 06:54:48 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Ср авг 14 06:54:48 2019]  ? mutex_lock+0x12/0x40
[Ср авг 14 06:54:48 2019]  zio_ready+0x49/0x430 [zfs]
[Ср авг 14 06:54:48 2019]  ? taskq_member+0x18/0x30 [spl]
[Ср авг 14 06:54:48 2019]  zio_execute+0x95/0xf0 [zfs]
[Ср авг 14 06:54:48 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Ср авг 14 06:54:48 2019]  ? wake_up_q+0x80/0x80
[Ср авг 14 06:54:48 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Ср авг 14 06:54:48 2019]  kthread+0x105/0x140
[Ср авг 14 06:54:48 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ret_from_fork+0x22/0x40
[Ср авг 14 06:54:48 2019] INFO: task z_wr_iss:2887 blocked for more than 120 seconds.
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] z_wr_iss        D    0  2887      2 0x80000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  ? avl_nearest+0x2b/0x30 [zavl]
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  rwsem_down_write_failed+0x208/0x390
[Ср авг 14 06:54:48 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Ср авг 14 06:54:48 2019]  call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Ср авг 14 06:54:48 2019]  down_write+0x2d/0x40
[Ср авг 14 06:54:48 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Ср авг 14 06:54:48 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Ср авг 14 06:54:48 2019]  ? mutex_lock+0x12/0x40
[Ср авг 14 06:54:48 2019]  zio_ready+0x49/0x430 [zfs]
[Ср авг 14 06:54:48 2019]  ? taskq_member+0x18/0x30 [spl]
[Ср авг 14 06:54:48 2019]  zio_execute+0x95/0xf0 [zfs]
[Ср авг 14 06:54:48 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Ср авг 14 06:54:48 2019]  ? wake_up_q+0x80/0x80
[Ср авг 14 06:54:48 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Ср авг 14 06:54:48 2019]  kthread+0x105/0x140
[Ср авг 14 06:54:48 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ret_from_fork+0x22/0x40
[Ср авг 14 06:54:48 2019] INFO: task z_wr_iss:2911 blocked for more than 120 seconds.
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] z_wr_iss        D    0  2911      2 0x80000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  ? avl_nearest+0x2b/0x30 [zavl]
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  rwsem_down_write_failed+0x208/0x390
[Ср авг 14 06:54:48 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Ср авг 14 06:54:48 2019]  call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Ср авг 14 06:54:48 2019]  down_write+0x2d/0x40
[Ср авг 14 06:54:48 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Ср авг 14 06:54:48 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Ср авг 14 06:54:48 2019]  ? mutex_lock+0x12/0x40
[Ср авг 14 06:54:48 2019]  zio_ready+0x49/0x430 [zfs]
[Ср авг 14 06:54:48 2019]  ? taskq_member+0x18/0x30 [spl]
[Ср авг 14 06:54:48 2019]  zio_execute+0x95/0xf0 [zfs]
[Ср авг 14 06:54:48 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Ср авг 14 06:54:48 2019]  ? wake_up_q+0x80/0x80
[Ср авг 14 06:54:48 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Ср авг 14 06:54:48 2019]  kthread+0x105/0x140
[Ср авг 14 06:54:48 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ret_from_fork+0x22/0x40
[Ср авг 14 06:54:48 2019] INFO: task z_wr_iss:2937 blocked for more than 120 seconds.
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] z_wr_iss        D    0  2937      2 0x80000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  ? avl_nearest+0x2b/0x30 [zavl]
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  rwsem_down_write_failed+0x208/0x390
[Ср авг 14 06:54:48 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Ср авг 14 06:54:48 2019]  call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Ср авг 14 06:54:48 2019]  down_write+0x2d/0x40
[Ср авг 14 06:54:48 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Ср авг 14 06:54:48 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Ср авг 14 06:54:48 2019]  ? mutex_lock+0x12/0x40
[Ср авг 14 06:54:48 2019]  zio_ready+0x49/0x430 [zfs]
[Ср авг 14 06:54:48 2019]  ? taskq_member+0x18/0x30 [spl]
[Ср авг 14 06:54:48 2019]  zio_execute+0x95/0xf0 [zfs]
[Ср авг 14 06:54:48 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Ср авг 14 06:54:48 2019]  ? wake_up_q+0x80/0x80
[Ср авг 14 06:54:48 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Ср авг 14 06:54:48 2019]  kthread+0x105/0x140
[Ср авг 14 06:54:48 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ret_from_fork+0x22/0x40
[Ср авг 14 06:54:48 2019] INFO: task z_wr_iss:2943 blocked for more than 120 seconds.
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] z_wr_iss        D    0  2943      2 0x80000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  ? avl_nearest+0x2b/0x30 [zavl]
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  rwsem_down_write_failed+0x208/0x390
[Ср авг 14 06:54:48 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Ср авг 14 06:54:48 2019]  call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Ср авг 14 06:54:48 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Ср авг 14 06:54:48 2019]  down_write+0x2d/0x40
[Ср авг 14 06:54:48 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Ср авг 14 06:54:48 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Ср авг 14 06:54:48 2019]  ? mutex_lock+0x12/0x40
[Ср авг 14 06:54:48 2019]  zio_ready+0x49/0x430 [zfs]
[Ср авг 14 06:54:48 2019]  ? taskq_member+0x18/0x30 [spl]
[Ср авг 14 06:54:48 2019]  zio_execute+0x95/0xf0 [zfs]
[Ср авг 14 06:54:48 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Ср авг 14 06:54:48 2019]  ? wake_up_q+0x80/0x80
[Ср авг 14 06:54:48 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Ср авг 14 06:54:48 2019]  kthread+0x105/0x140
[Ср авг 14 06:54:48 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ret_from_fork+0x22/0x40
[Ср авг 14 06:54:48 2019] INFO: task txg_sync:4545 blocked for more than 120 seconds.
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] txg_sync        D    0  4545      2 0x80000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  io_schedule+0x16/0x40
[Ср авг 14 06:54:48 2019]  cv_wait_common+0xb2/0x140 [spl]
[Ср авг 14 06:54:48 2019]  ? wait_woken+0x80/0x80
[Ср авг 14 06:54:48 2019]  __cv_wait_io+0x18/0x20 [spl]
[Ср авг 14 06:54:48 2019]  zio_wait+0x105/0x1b0 [zfs]
[Ср авг 14 06:54:48 2019]  dsl_pool_sync+0xb8/0x430 [zfs]
[Ср авг 14 06:54:48 2019]  spa_sync+0x42d/0xd50 [zfs]
[Ср авг 14 06:54:48 2019]  txg_sync_thread+0x2d4/0x4a0 [zfs]
[Ср авг 14 06:54:48 2019]  ? txg_quiesce_thread+0x3f0/0x3f0 [zfs]
[Ср авг 14 06:54:48 2019]  thread_generic_wrapper+0x74/0x90 [spl]
[Ср авг 14 06:54:48 2019]  kthread+0x105/0x140
[Ср авг 14 06:54:48 2019]  ? __thread_exit+0x20/0x20 [spl]
[Ср авг 14 06:54:48 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Ср авг 14 06:54:48 2019]  ret_from_fork+0x22/0x40
[Ср авг 14 06:54:48 2019] INFO: task kvm:3873355 blocked for more than 120 seconds.
[Ср авг 14 06:54:48 2019]       Tainted: P           O     4.15.18-9-pve #1
[Ср авг 14 06:54:48 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Ср авг 14 06:54:48 2019] kvm             D    0 3873355      1 0x00000000
[Ср авг 14 06:54:48 2019] Call Trace:
[Ср авг 14 06:54:48 2019]  __schedule+0x3e0/0x870
[Ср авг 14 06:54:48 2019]  schedule+0x36/0x80
[Ср авг 14 06:54:48 2019]  io_schedule+0x16/0x40
[Ср авг 14 06:54:48 2019]  wait_on_page_bit_common+0xf3/0x190
[Ср авг 14 06:54:48 2019]  ? page_cache_tree_insert+0xe0/0xe0
[Ср авг 14 06:54:48 2019]  __filemap_fdatawait_range+0xfa/0x160
[Ср авг 14 06:54:48 2019]  file_write_and_wait_range+0x70/0xb0
[Ср авг 14 06:54:48 2019]  blkdev_fsync+0x1b/0x50
[Ср авг 14 06:54:48 2019]  vfs_fsync_range+0x51/0xb0
[Ср авг 14 06:54:48 2019]  do_fsync+0x3d/0x70
[Ср авг 14 06:54:48 2019]  SyS_fdatasync+0x13/0x20
[Ср авг 14 06:54:48 2019]  do_syscall_64+0x73/0x130
[Ср авг 14 06:54:48 2019]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[Ср авг 14 06:54:48 2019] RIP: 0033:0x7fb7c7e0960d
[Ср авг 14 06:54:48 2019] RSP: 002b:00007fb56cbfc0b0 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
[Ср авг 14 06:54:48 2019] RAX: ffffffffffffffda RBX: 00007fb5a657fab0 RCX: 00007fb7c7e0960d
[Ср авг 14 06:54:48 2019] RDX: 00007fb7b148fc90 RSI: 0000000000000000 RDI: 000000000000001b
[Ср авг 14 06:54:48 2019] RBP: 00000000fffffffb R08: 0000000000000000 R09: 00000000ffffffff
[Ср авг 14 06:54:48 2019] R10: 00007fb56cbfc1a0 R11: 0000000000000293 R12: 00007fb7b16c5e80
[Ср авг 14 06:54:48 2019] R13: 00007fb7b148fcf8 R14: 00007fb5a9375320 R15: 00007fb7ce125040

[Чт авг 15 00:09:31 2019] INFO: task zvol:2406 blocked for more than 120 seconds.
--
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] zvol            D    0  2406      2 0x80000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  ? __switch_to_asm+0x34/0x70
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  rwsem_down_read_failed+0x10a/0x170
[Чт авг 15 00:09:31 2019]  ? _cond_resched+0x1a/0x50
[Чт авг 15 00:09:31 2019]  call_rwsem_down_read_failed+0x18/0x30
[Чт авг 15 00:09:31 2019]  ? mutex_lock+0x12/0x40
[Чт авг 15 00:09:31 2019]  ? call_rwsem_down_read_failed+0x18/0x30
[Чт авг 15 00:09:31 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 00:09:31 2019]  down_read+0x20/0x40
[Чт авг 15 00:09:31 2019]  dbuf_read+0x2fe/0x930 [zfs]
[Чт авг 15 00:09:31 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 00:09:31 2019]  ? dbuf_hold_impl+0xb1/0xd0 [zfs]
[Чт авг 15 00:09:31 2019]  dmu_tx_check_ioerr+0x68/0xc0 [zfs]
[Чт авг 15 00:09:31 2019]  dmu_tx_count_write+0xed/0x1b0 [zfs]
[Чт авг 15 00:09:31 2019]  dmu_tx_hold_write+0x41/0x60 [zfs]
[Чт авг 15 00:09:31 2019]  zvol_write+0x179/0x640 [zfs]
[Чт авг 15 00:09:31 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 00:09:31 2019]  ? wake_up_q+0x80/0x80
[Чт авг 15 00:09:31 2019]  kthread+0x105/0x140
[Чт авг 15 00:09:31 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 00:09:31 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 00:09:31 2019]  ? do_syscall_64+0x73/0x130
[Чт авг 15 00:09:31 2019]  ? SyS_exit_group+0x14/0x20
[Чт авг 15 00:09:31 2019]  ret_from_fork+0x22/0x40
[Чт авг 15 00:09:31 2019] INFO: task z_wr_iss:2823 blocked for more than 120 seconds.
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] z_wr_iss        D    0  2823      2 0x80000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 00:09:31 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 00:09:31 2019]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 00:09:31 2019]  down_write+0x2d/0x40
[Чт авг 15 00:09:31 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 00:09:31 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 00:09:31 2019]  ? mutex_lock+0x12/0x40
[Чт авг 15 00:09:31 2019]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 00:09:31 2019]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 00:09:31 2019]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 00:09:31 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 00:09:31 2019]  ? wake_up_q+0x80/0x80
[Чт авг 15 00:09:31 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 00:09:31 2019]  kthread+0x105/0x140
[Чт авг 15 00:09:31 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 00:09:31 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 00:09:31 2019]  ret_from_fork+0x22/0x40
[Чт авг 15 00:09:31 2019] INFO: task z_wr_iss:2826 blocked for more than 120 seconds.
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] z_wr_iss        D    0  2826      2 0x80000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 00:09:31 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 00:09:31 2019]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  down_write+0x2d/0x40
[Чт авг 15 00:09:31 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 00:09:31 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 00:09:31 2019]  ? mutex_lock+0x12/0x40
[Чт авг 15 00:09:31 2019]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 00:09:31 2019]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 00:09:31 2019]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 00:09:31 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 00:09:31 2019]  ? wake_up_q+0x80/0x80
[Чт авг 15 00:09:31 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 00:09:31 2019]  kthread+0x105/0x140
[Чт авг 15 00:09:31 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 00:09:31 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 00:09:31 2019]  ret_from_fork+0x22/0x40
[Чт авг 15 00:09:31 2019] INFO: task z_wr_iss:2859 blocked for more than 120 seconds.
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] z_wr_iss        D    0  2859      2 0x80000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 00:09:31 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 00:09:31 2019]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 00:09:31 2019]  down_write+0x2d/0x40
[Чт авг 15 00:09:31 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 00:09:31 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 00:09:31 2019]  ? mutex_lock+0x12/0x40
[Чт авг 15 00:09:31 2019]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 00:09:31 2019]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 00:09:31 2019]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 00:09:31 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 00:09:31 2019]  ? wake_up_q+0x80/0x80
[Чт авг 15 00:09:31 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 00:09:31 2019]  kthread+0x105/0x140
[Чт авг 15 00:09:31 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 00:09:31 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 00:09:31 2019]  ret_from_fork+0x22/0x40
[Чт авг 15 00:09:31 2019] INFO: task z_wr_iss:2869 blocked for more than 120 seconds.
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] z_wr_iss        D    0  2869      2 0x80000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 00:09:31 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 00:09:31 2019]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 00:09:31 2019]  down_write+0x2d/0x40
[Чт авг 15 00:09:31 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 00:09:31 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 00:09:31 2019]  ? mutex_lock+0x12/0x40
[Чт авг 15 00:09:31 2019]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 00:09:31 2019]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 00:09:31 2019]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 00:09:31 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 00:09:31 2019]  ? wake_up_q+0x80/0x80
[Чт авг 15 00:09:31 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 00:09:31 2019]  kthread+0x105/0x140
[Чт авг 15 00:09:31 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 00:09:31 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 00:09:31 2019]  ret_from_fork+0x22/0x40
[Чт авг 15 00:09:31 2019] INFO: task z_wr_iss:2872 blocked for more than 120 seconds.
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] z_wr_iss        D    0  2872      2 0x80000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 00:09:31 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 00:09:31 2019]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 00:09:31 2019]  down_write+0x2d/0x40
[Чт авг 15 00:09:31 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 00:09:31 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 00:09:31 2019]  ? mutex_lock+0x12/0x40
[Чт авг 15 00:09:31 2019]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 00:09:31 2019]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 00:09:31 2019]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 00:09:31 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 00:09:31 2019]  ? wake_up_q+0x80/0x80
[Чт авг 15 00:09:31 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 00:09:31 2019]  kthread+0x105/0x140
[Чт авг 15 00:09:31 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 00:09:31 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 00:09:31 2019]  ret_from_fork+0x22/0x40
[Чт авг 15 00:09:31 2019] INFO: task z_wr_iss:2883 blocked for more than 120 seconds.
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] z_wr_iss        D    0  2883      2 0x80000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 00:09:31 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 00:09:31 2019]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  down_write+0x2d/0x40
[Чт авг 15 00:09:31 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 00:09:31 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 00:09:31 2019]  ? mutex_lock+0x12/0x40
[Чт авг 15 00:09:31 2019]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 00:09:31 2019]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 00:09:31 2019]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 00:09:31 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 00:09:31 2019]  ? wake_up_q+0x80/0x80
[Чт авг 15 00:09:31 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 00:09:31 2019]  kthread+0x105/0x140
[Чт авг 15 00:09:31 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 00:09:31 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 00:09:31 2019]  ret_from_fork+0x22/0x40
[Чт авг 15 00:09:31 2019] INFO: task z_wr_iss:2896 blocked for more than 120 seconds.
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] z_wr_iss        D    0  2896      2 0x80000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 00:09:31 2019]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 00:09:31 2019]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 00:09:31 2019]  down_write+0x2d/0x40
[Чт авг 15 00:09:31 2019]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 00:09:31 2019]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 00:09:31 2019]  ? mutex_lock+0x12/0x40
[Чт авг 15 00:09:31 2019]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 00:09:31 2019]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 00:09:31 2019]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 00:09:31 2019]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 00:09:31 2019]  ? wake_up_q+0x80/0x80
[Чт авг 15 00:09:31 2019]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 00:09:31 2019]  kthread+0x105/0x140
[Чт авг 15 00:09:31 2019]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 00:09:31 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 00:09:31 2019]  ret_from_fork+0x22/0x40
[Чт авг 15 00:09:31 2019] INFO: task txg_sync:4452 blocked for more than 120 seconds.
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] txg_sync        D    0  4452      2 0x80000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  io_schedule+0x16/0x40
[Чт авг 15 00:09:31 2019]  cv_wait_common+0xb2/0x140 [spl]
[Чт авг 15 00:09:31 2019]  ? wait_woken+0x80/0x80
[Чт авг 15 00:09:31 2019]  __cv_wait_io+0x18/0x20 [spl]
[Чт авг 15 00:09:31 2019]  zio_wait+0x105/0x1b0 [zfs]
[Чт авг 15 00:09:31 2019]  dsl_pool_sync+0xb8/0x430 [zfs]
[Чт авг 15 00:09:31 2019]  spa_sync+0x42d/0xd50 [zfs]
[Чт авг 15 00:09:31 2019]  txg_sync_thread+0x2d4/0x4a0 [zfs]
[Чт авг 15 00:09:31 2019]  ? txg_quiesce_thread+0x3f0/0x3f0 [zfs]
[Чт авг 15 00:09:31 2019]  thread_generic_wrapper+0x74/0x90 [spl]
[Чт авг 15 00:09:31 2019]  kthread+0x105/0x140
[Чт авг 15 00:09:31 2019]  ? __thread_exit+0x20/0x20 [spl]
[Чт авг 15 00:09:31 2019]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 00:09:31 2019]  ret_from_fork+0x22/0x40
[Чт авг 15 00:09:31 2019] INFO: task kvm:18830 blocked for more than 120 seconds.
[Чт авг 15 00:09:31 2019]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 00:09:31 2019] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 00:09:31 2019] kvm             D    0 18830      1 0x00000000
[Чт авг 15 00:09:31 2019] Call Trace:
[Чт авг 15 00:09:31 2019]  __schedule+0x3e0/0x870
[Чт авг 15 00:09:31 2019]  schedule+0x36/0x80
[Чт авг 15 00:09:31 2019]  io_schedule+0x16/0x40
[Чт авг 15 00:09:31 2019]  wait_on_page_bit_common+0xf3/0x190
[Чт авг 15 00:09:31 2019]  ? page_cache_tree_insert+0xe0/0xe0
[Чт авг 15 00:09:31 2019]  __filemap_fdatawait_range+0xfa/0x160
[Чт авг 15 00:09:31 2019]  file_write_and_wait_range+0x70/0xb0
[Чт авг 15 00:09:31 2019]  blkdev_fsync+0x1b/0x50
[Чт авг 15 00:09:31 2019]  vfs_fsync_range+0x51/0xb0
[Чт авг 15 00:09:31 2019]  do_fsync+0x3d/0x70
[Чт авг 15 00:09:31 2019]  SyS_fdatasync+0x13/0x20
[Чт авг 15 00:09:31 2019]  do_syscall_64+0x73/0x130
[Чт авг 15 00:09:31 2019]  entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[Чт авг 15 00:09:31 2019] RIP: 0033:0x7f5d54d8560d
[Чт авг 15 00:09:31 2019] RSP: 002b:00007f5b08bfc0b0 EFLAGS: 00000293 ORIG_RAX: 000000000000004b
[Чт авг 15 00:09:31 2019] RAX: ffffffffffffffda RBX: 00007f5b362172e0 RCX: 00007f5d54d8560d
[Чт авг 15 00:09:31 2019] RDX: 00007f5d3e48fc90 RSI: 0000000000000000 RDI: 000000000000001b
[Чт авг 15 00:09:31 2019] RBP: 00000000fffffffb R08: 0000000000000000 R09: 00000000ffffffff
[Чт авг 15 00:09:31 2019] R10: 00007f5b08bfc1a0 R11: 0000000000000293 R12: 00007f5d3e6c5e80
[Чт авг 15 00:09:31 2019] R13: 00007f5d3e48fcf8 R14: 00007f5b3637da20 R15: 00007f5d5b0a1040
[Чт авг 15 08:40:27 2019 <   86,203341>] INFO: task zvol:2384 blocked for more than 120 seconds.
--
[Чт авг 15 08:40:27 2019 <    0,000038>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000021>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000025>] zvol            D    0  2384      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000004>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000015>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? __switch_to_asm+0x34/0x70
[Чт авг 15 08:40:27 2019 <    0,000002>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000002>]  rwsem_down_read_failed+0x10a/0x170
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? _cond_resched+0x1a/0x50
[Чт авг 15 08:40:27 2019 <    0,000003>]  call_rwsem_down_read_failed+0x18/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? call_rwsem_down_read_failed+0x18/0x30
[Чт авг 15 08:40:27 2019 <    0,000009>]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 08:40:27 2019 <    0,000004>]  down_read+0x20/0x40
[Чт авг 15 08:40:27 2019 <    0,000029>]  dbuf_read+0x2fe/0x930 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 08:40:27 2019 <    0,000023>]  ? dbuf_hold_impl+0xb1/0xd0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000028>]  dmu_tx_check_ioerr+0x68/0xc0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000027>]  dmu_tx_count_write+0xed/0x1b0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000026>]  dmu_tx_hold_write+0x41/0x60 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000033>]  zvol_write+0x179/0x640 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000005>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? do_syscall_64+0x73/0x130
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? SyS_exit_group+0x14/0x20
[Чт авг 15 08:40:27 2019 <    0,000002>]  ret_from_fork+0x22/0x40
[Чт авг 15 08:40:27 2019 <    0,000010>] INFO: task z_wr_iss:2873 blocked for more than 120 seconds.
[Чт авг 15 08:40:27 2019 <    0,000023>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000019>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000024>] z_wr_iss        D    0  2873      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000002>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000002>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? avl_nearest+0x2b/0x30 [zavl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000002>]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 08:40:27 2019 <    0,000033>]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  down_write+0x2d/0x40
[Чт авг 15 08:40:27 2019 <    0,000023>]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000022>]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000031>]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 08:40:27 2019 <    0,000031>]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000030>]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000002>]  ret_from_fork+0x22/0x40
[Чт авг 15 08:40:27 2019 <    0,000002>] INFO: task z_wr_iss:2874 blocked for more than 120 seconds.
[Чт авг 15 08:40:27 2019 <    0,000022>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000019>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000024>] z_wr_iss        D    0  2874      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000002>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000001>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? avl_nearest+0x2b/0x30 [zavl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000001>]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 08:40:27 2019 <    0,000031>]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  down_write+0x2d/0x40
[Чт авг 15 08:40:27 2019 <    0,000022>]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000021>]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000028>]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 08:40:27 2019 <    0,000029>]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000029>]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000001>]  ret_from_fork+0x22/0x40
[Чт авг 15 08:40:27 2019 <    0,000003>] INFO: task z_wr_iss:2879 blocked for more than 120 seconds.
[Чт авг 15 08:40:27 2019 <    0,000022>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000019>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000024>] z_wr_iss        D    0  2879      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000002>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000001>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? avl_nearest+0x2b/0x30 [zavl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000002>]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 08:40:27 2019 <    0,000029>]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  down_write+0x2d/0x40
[Чт авг 15 08:40:27 2019 <    0,000021>]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000020>]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000028>]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 08:40:27 2019 <    0,000027>]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000027>]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000002>]  ret_from_fork+0x22/0x40
[Чт авг 15 08:40:27 2019 <    0,000005>] INFO: task z_wr_iss:2891 blocked for more than 120 seconds.
[Чт авг 15 08:40:27 2019 <    0,000023>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000019>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000025>] z_wr_iss        D    0  2891      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000001>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000002>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? avl_nearest+0x2b/0x30 [zavl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000001>]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 08:40:27 2019 <    0,000029>]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  down_write+0x2d/0x40
[Чт авг 15 08:40:27 2019 <    0,000021>]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000021>]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000027>]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 08:40:27 2019 <    0,000027>]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000027>]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000001>]  ret_from_fork+0x22/0x40
[Чт авг 15 08:40:27 2019 <    0,000008>] INFO: task z_wr_iss:2925 blocked for more than 120 seconds.
[Чт авг 15 08:40:27 2019 <    0,000023>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000019>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000024>] z_wr_iss        D    0  2925      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000001>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000002>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000002>]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 08:40:27 2019 <    0,000028>]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  down_write+0x2d/0x40
[Чт авг 15 08:40:27 2019 <    0,000021>]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000021>]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000027>]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 08:40:27 2019 <    0,000027>]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000027>]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000002>]  ret_from_fork+0x22/0x40
[Чт авг 15 08:40:27 2019 <    0,000003>] INFO: task z_wr_iss:2930 blocked for more than 120 seconds.
[Чт авг 15 08:40:27 2019 <    0,000022>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000019>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000024>] z_wr_iss        D    0  2930      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000002>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000002>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? avl_insert+0xbb/0xd0 [zavl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000001>]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 08:40:27 2019 <    0,000028>]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  down_write+0x2d/0x40
[Чт авг 15 08:40:27 2019 <    0,000021>]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000021>]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000027>]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 08:40:27 2019 <    0,000026>]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000026>]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000002>]  ret_from_fork+0x22/0x40
[Чт авг 15 08:40:27 2019 <    0,000002>] INFO: task z_wr_iss:2935 blocked for more than 120 seconds.
[Чт авг 15 08:40:27 2019 <    0,000022>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000019>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000024>] z_wr_iss        D    0  2935      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000002>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000001>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? avl_insert+0xbb/0xd0 [zavl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000001>]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 08:40:27 2019 <    0,000028>]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  down_write+0x2d/0x40
[Чт авг 15 08:40:27 2019 <    0,000021>]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000021>]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000027>]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 08:40:27 2019 <    0,000026>]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000026>]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000002>]  ret_from_fork+0x22/0x40
[Чт авг 15 08:40:27 2019 <    0,000001>] INFO: task z_wr_iss:2936 blocked for more than 120 seconds.
[Чт авг 15 08:40:27 2019 <    0,000022>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000018>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000025>] z_wr_iss        D    0  2936      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000001>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000002>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? avl_insert+0xbb/0xd0 [zavl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000001>]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 08:40:27 2019 <    0,000028>]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  down_write+0x2d/0x40
[Чт авг 15 08:40:27 2019 <    0,000021>]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000020>]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000027>]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 08:40:27 2019 <    0,000026>]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000027>]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000002>]  ret_from_fork+0x22/0x40
[Чт авг 15 08:40:27 2019 <    0,000007>] INFO: task z_wr_iss:2962 blocked for more than 120 seconds.
[Чт авг 15 08:40:27 2019 <    0,000021>]       Tainted: P           O     4.15.18-9-pve #1
[Чт авг 15 08:40:27 2019 <    0,000019>] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[Чт авг 15 08:40:27 2019 <    0,000024>] z_wr_iss        D    0  2962      2 0x80000000
[Чт авг 15 08:40:27 2019 <    0,000002>] Call Trace:
[Чт авг 15 08:40:27 2019 <    0,000001>]  __schedule+0x3e0/0x870
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? avl_nearest+0x2b/0x30 [zavl]
[Чт авг 15 08:40:27 2019 <    0,000001>]  schedule+0x36/0x80
[Чт авг 15 08:40:27 2019 <    0,000002>]  rwsem_down_write_failed+0x208/0x390
[Чт авг 15 08:40:27 2019 <    0,000027>]  ? spa_config_exit+0x7f/0xb0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? call_rwsem_down_write_failed+0x17/0x30
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? spl_kmem_free+0x33/0x40 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  down_write+0x2d/0x40
[Чт авг 15 08:40:27 2019 <    0,000021>]  dbuf_write_ready+0x1ce/0x2d0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000020>]  arc_write_ready+0x9e/0x320 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000001>]  ? mutex_lock+0x12/0x40
[Чт авг 15 08:40:27 2019 <    0,000027>]  zio_ready+0x49/0x430 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_member+0x18/0x30 [spl]
[Чт авг 15 08:40:27 2019 <    0,000026>]  zio_execute+0x95/0xf0 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000004>]  taskq_thread+0x2ae/0x4d0 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? wake_up_q+0x80/0x80
[Чт авг 15 08:40:27 2019 <    0,000026>]  ? zio_reexecute+0x390/0x390 [zfs]
[Чт авг 15 08:40:27 2019 <    0,000002>]  kthread+0x105/0x140
[Чт авг 15 08:40:27 2019 <    0,000003>]  ? taskq_thread_should_stop+0x70/0x70 [spl]
[Чт авг 15 08:40:27 2019 <    0,000002>]  ? kthread_create_worker_on_cpu+0x70/0x70
[Чт авг 15 08:40:27 2019 <    0,000001>]  ret_from_fork+0x22/0x40

image image image image

Temporary solve

Reboot (by reset system or sysrq-trigger) Move of one zfs volume to another host Disable compression on affected volume

GregorKopka commented 5 years ago

@behlendorf shouldn't this get the defect instead of performance label?

behlendorf commented 5 years ago

@GregorKopka indeed, this does look like a possible deadlock on a dbuf. It would be useful to know if this can be reproduced in master since some of the relevant locks have been split up.

bgly commented 4 years ago

I actually also get these hangs with 0.8.2. 

Dec 23 06:31:20 spplusc-6 kernel: INFO: task nfsd:6469 blocked for more than 900 seconds.
Dec 23 06:31:20 spplusc-6 kernel:      Tainted: P           OE     4.19.65-1c.el7.x86_64 #1
Dec 23 06:31:20 spplusc-6 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 23 06:31:20 spplusc-6 kernel: nfsd            D    0  6469      2 0x80000080
Dec 23 06:31:20 spplusc-6 kernel: Call Trace:
Dec 23 06:31:20 spplusc-6 kernel: ? __schedule+0x2ab/0x880
Dec 23 06:31:20 spplusc-6 kernel: schedule+0x32/0x80
Dec 23 06:31:20 spplusc-6 kernel: rwsem_down_read_failed+0x139/0x1c0
Dec 23 06:31:20 spplusc-6 kernel: call_rwsem_down_read_failed+0x14/0x30
Dec 23 06:31:20 spplusc-6 kernel: down_read+0x1c/0x30
Dec 23 06:31:20 spplusc-6 kernel: dmu_buf_lock_parent+0x5c/0xd0 [zfs]
Dec 23 06:31:20 spplusc-6 kernel: ? _cond_resched+0x15/0x30
Dec 23 06:31:20 spplusc-6 kernel: ? down_read+0xe/0x30
Dec 23 06:31:20 spplusc-6 kernel: dbuf_dirty+0x300/0x780 [zfs]
Dec 23 06:31:20 spplusc-6 kernel: dmu_write_uio_dnode+0x70/0x140 [zfs]
Dec 23 06:31:20 spplusc-6 kernel: dmu_write_uio_dbuf+0x4e/0x70 [zfs]
Dec 23 06:31:20 spplusc-6 kernel: zfs_write+0xb4e/0xcd0 [zfs]
Dec 23 06:31:20 spplusc-6 kernel: ? iput+0x6f/0x1d0
Dec 23 06:31:20 spplusc-6 kernel: ? __d_obtain_alias+0x32/0x80
Dec 23 06:31:20 spplusc-6 kernel: zpl_write_common_iovec+0xa9/0x120 [zfs]
Dec 23 06:31:20 spplusc-6 kernel: zpl_iter_write_common+0x98/0xc0 [zfs]
Dec 23 06:31:20 spplusc-6 kernel: zpl_iter_write+0x3f/0x70 [zfs]
Dec 23 06:31:20 spplusc-6 kernel: ? selinux_file_permission+0xe1/0x130
Dec 23 06:31:20 spplusc-6 kernel: do_iter_readv_writev+0x132/0x1b0
Dec 23 06:31:20 spplusc-6 kernel: do_iter_write+0x78/0x180
Dec 23 06:31:20 spplusc-6 kernel: nfsd_vfs_write+0xff/0x470 [nfsd]
Dec 23 06:31:20 spplusc-6 kernel: nfsd_write+0x94/0x180 [nfsd]
Dec 23 06:31:20 spplusc-6 kernel: nfsd3_proc_write+0x106/0x180 [nfsd]
Dec 23 06:31:20 spplusc-6 kernel: nfsd_dispatch+0xb7/0x250 [nfsd]
Dec 23 06:31:20 spplusc-6 kernel: svc_process_common+0x39e/0x800 [sunrpc]
Dec 23 06:31:20 spplusc-6 kernel: svc_process+0xeb/0x100 [sunrpc]
Dec 23 06:31:20 spplusc-6 kernel: nfsd+0xe3/0x150 [nfsd]
Dec 23 06:31:20 spplusc-6 kernel: kthread+0xf8/0x130
Dec 23 06:31:20 spplusc-6 kernel: ? nfsd_destroy+0x60/0x60 [nfsd]
Dec 23 06:31:20 spplusc-6 kernel: ? kthread_bind+0x10/0x10
Dec 23 06:31:20 spplusc-6 kernel: ret_from_fork+0x35/0x40
bgly commented 4 years ago

Mar 1 20:47:38 spvss-ptdca1 kernel: INFO: task nfsd:8218 blocked for more than 900 seconds. Mar 1 20:47:38 spvss-ptdca1 kernel: Tainted: P OE 4.19.101-1c.el7.x86_64 #1 Mar 1 20:47:38 spvss-ptdca1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd D 0 8218 2 0x80000080 Mar 1 20:47:38 spvss-ptdca1 kernel: Call Trace: Mar 1 20:47:38 spvss-ptdca1 kernel: ? __schedule+0x2ab/0x880 Mar 1 20:47:38 spvss-ptdca1 kernel: schedule+0x32/0x80 Mar 1 20:47:38 spvss-ptdca1 kernel: rwsem_down_read_failed+0x139/0x1c0 Mar 1 20:47:38 spvss-ptdca1 kernel: call_rwsem_down_read_failed+0x14/0x30 Mar 1 20:47:38 spvss-ptdca1 kernel: down_read+0x1c/0x30 Mar 1 20:47:38 spvss-ptdca1 kernel: dbuf_hold_impl+0x517/0x590 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dbuf_hold_level+0x33/0x60 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dmu_buf_hold_array_by_dnode+0xdc/0x4a0 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dmu_write_uio_dnode+0x56/0x140 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: ? dmu_tx_try_assign+0x304/0x370 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dmu_write_uio_dbuf+0x4e/0x70 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: zfs_write+0xb4e/0xcd0 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: zpl_write_common_iovec+0xa9/0x120 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: zpl_iter_write_common+0x98/0xc0 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: zpl_iter_write+0x3f/0x70 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: ? selinux_file_permission+0xe1/0x130 Mar 1 20:47:38 spvss-ptdca1 kernel: do_iter_readv_writev+0x132/0x1b0 Mar 1 20:47:38 spvss-ptdca1 kernel: do_iter_write+0x78/0x180 Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd_vfs_write+0xff/0x470 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd_write+0x94/0x180 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd3_proc_write+0x106/0x180 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd_dispatch+0xb7/0x250 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: svc_process_common+0x39e/0x800 [sunrpc] Mar 1 20:47:38 spvss-ptdca1 kernel: svc_process+0xeb/0x100 [sunrpc] Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd+0xe3/0x150 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: kthread+0xf8/0x130 Mar 1 20:47:38 spvss-ptdca1 kernel: ? nfsd_destroy+0x60/0x60 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: ? kthread_bind+0x10/0x10 Mar 1 20:47:38 spvss-ptdca1 kernel: ret_from_fork+0x35/0x40 Mar 1 20:47:38 spvss-ptdca1 kernel: INFO: task nfsd:8219 blocked for more than 900 seconds. Mar 1 20:47:38 spvss-ptdca1 kernel: Tainted: P OE 4.19.101-1c.el7.x86_64 #1 Mar 1 20:47:38 spvss-ptdca1 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd D 0 8219 2 0x80000080 Mar 1 20:47:38 spvss-ptdca1 kernel: Call Trace: Mar 1 20:47:38 spvss-ptdca1 kernel: ? schedule+0x2ab/0x880 Mar 1 20:47:38 spvss-ptdca1 kernel: schedule+0x32/0x80 Mar 1 20:47:38 spvss-ptdca1 kernel: schedule_preempt_disabled+0xa/0x10 Mar 1 20:47:38 spvss-ptdca1 kernel: mutex_lock.isra.11+0x21b/0x4e0 Mar 1 20:47:38 spvss-ptdca1 kernel: ? cityhash4+0x78/0xa0 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dbuf_find+0xb8/0x190 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dbuf_hold_impl+0x62/0x590 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dbuf_hold_level+0x33/0x60 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dmu_tx_check_ioerr+0x32/0xc0 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dmu_tx_count_write+0xdd/0x190 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: dmu_tx_hold_write_by_dnode+0x35/0x50 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: zfs_write+0x516/0xcd0 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: zpl_write_common_iovec+0xa9/0x120 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: zpl_iter_write_common+0x98/0xc0 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: zpl_iter_write+0x3f/0x70 [zfs] Mar 1 20:47:38 spvss-ptdca1 kernel: ? selinux_file_permission+0xe1/0x130 Mar 1 20:47:38 spvss-ptdca1 kernel: do_iter_readv_writev+0x132/0x1b0 Mar 1 20:47:38 spvss-ptdca1 kernel: do_iter_write+0x78/0x180 Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd_vfs_write+0xff/0x470 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd_write+0x94/0x180 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd3_proc_write+0x106/0x180 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd_dispatch+0xb7/0x250 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: svc_process_common+0x39e/0x800 [sunrpc] Mar 1 20:47:38 spvss-ptdca1 kernel: svc_process+0xeb/0x100 [sunrpc] Mar 1 20:47:38 spvss-ptdca1 kernel: nfsd+0xe3/0x150 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: kthread+0xf8/0x130 Mar 1 20:47:38 spvss-ptdca1 kernel: ? nfsd_destroy+0x60/0x60 [nfsd] Mar 1 20:47:38 spvss-ptdca1 kernel: ? kthread_bind+0x10/0x10 Mar 1 20:47:38 spvss-ptdca1 kernel: ret_from_fork+0x35/0x40

Hit this again today and all I/O was hung afterwards. Also to note using 0.8.3 code with kernel that was released a few weeks ago.

glztmf commented 4 years ago

@bgly , is this issue actual to you? Did you somehow get around these hangs?

bgly commented 4 years ago

@glztmf I still experience these, sometimes more prevalent than others. Do you have the same issue?

glztmf commented 4 years ago

@bgly , i had the same issue with zfs 0.8.1 (pve 6) and remained on zfs 0.7.x

stale[bot] commented 3 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

GregorKopka commented 3 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.

@behlendorf should the stale bot not ignore issues marked as 'defect' ?

stale[bot] commented 1 year ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.