search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.65k stars 1.75k forks source link

INFO: task txg_sync:1211 blocked for more than 122 seconds. #11401

Closed reactormonk closed 2 years ago

reactormonk commented 3 years ago

System information

Type Version/Name
Distribution Name Archlinux
Distribution Version 2.0.0_5.9.14.arch1.1-1
Linux Kernel 5.9.14.arch1.1
Architecture x86_64
ZFS Version 2.0.0
SPL Version 2.0.0

Describe the problem you're observing

Disk IO slowdown (from 25MB/s steam download to ~5-10MB/s)

Describe how to reproduce the problem

Put some load on the system (currently mirror HDDs)

Include any warning/errors/backtraces from the system logs

INFO: task txg_sync:1211 blocked for more than 122 seconds.
Dec 24 20:52:43 exia kernel:       Tainted: P        W  OE     5.9.14-arch1-1 #1
Dec 24 20:52:43 exia kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Dec 24 20:52:43 exia kernel: task:txg_sync        state:D stack:    0 pid: 1211 ppid:     2 flags:0x00004080
Dec 24 20:52:43 exia kernel: Call Trace:
Dec 24 20:52:43 exia kernel:  __schedule+0x292/0x830
Dec 24 20:52:43 exia kernel:  schedule+0x46/0xf0
Dec 24 20:52:43 exia kernel:  schedule_timeout+0x99/0x170
Dec 24 20:52:43 exia kernel:  ? __next_timer_interrupt+0x100/0x100
Dec 24 20:52:43 exia kernel:  io_schedule_timeout+0x4c/0x80
Dec 24 20:52:43 exia kernel:  __cv_timedwait_common+0x129/0x160 [spl]
Dec 24 20:52:43 exia kernel:  ? wait_woken+0x80/0x80
Dec 24 20:52:43 exia kernel:  __cv_timedwait_io+0x15/0x20 [spl]
Dec 24 20:52:43 exia kernel:  zio_wait+0x13b/0x2d0 [zfs]
Dec 24 20:52:43 exia kernel:  dsl_pool_sync+0xd0/0x4c0 [zfs]
Dec 24 20:52:43 exia kernel:  spa_sync+0x56d/0xfc0 [zfs]
Dec 24 20:52:43 exia kernel:  ? _raw_spin_lock+0x13/0x30
Dec 24 20:52:43 exia kernel:  ? spa_txg_history_init_io+0x112/0x120 [zfs]
Dec 24 20:52:43 exia kernel:  txg_sync_thread+0x2dc/0x4a0 [zfs]
Dec 24 20:52:43 exia kernel:  ? txg_register_callbacks+0xa0/0xa0 [zfs]
Dec 24 20:52:43 exia kernel:  ? __thread_exit+0x20/0x20 [spl]
Dec 24 20:52:43 exia kernel:  thread_generic_wrapper+0x6f/0x80 [spl]
Dec 24 20:52:43 exia kernel:  kthread+0x142/0x160
Dec 24 20:52:43 exia kernel:  ? __kthread_bind_mask+0x60/0x60
Dec 24 20:52:43 exia kernel:  ret_from_fork+0x22/0x30
sempervictus commented 3 years ago

@reactormonk: any chance you're running PDS or BMQ schedulers? Pretty sure i've run into that exact problem under the old BMQ/PDS (they're separate in 5.9 now) on an 8 or 16 core system. What do the underlying disks look like in iostat when this is happening? Are there lots of requests and a full load, or does it look like the disks are at 100% utilization with relatively few requests going in/out of them? Also, how's steam being run? IIRC it needs lib32 so pristine 64b arch users often use flatpak or the like which can introduce overlayfs into the mix resulting in a really fun set of interactions between two bastard file systems (overlay and zfs dont always play nice for inotify and such without patching).

reactormonk commented 3 years ago

I got these kernel lines: amd_iommu=on iommu=pt.

reactormonk commented 3 years ago

Happened again:

                                                               capacity     operations     bandwidth 
pool                                                         alloc   free   read  write   read  write
-----------------------------------------------------------  -----  -----  -----  -----  -----  -----
pool                                                         4.26T  3.01T      1     60  17.4K  7.67M
  mirror                                                     4.26T  3.01T      1     60  17.4K  7.67M
    ata-ST8000DM004-2CX188_WCT3EZRV                              -      -      0     31  8.90K  3.84M
    ata-ST8000DM004-2CX188_WCT3EZE0                              -      -      0     29  8.50K  3.84M
cache                                                            -      -      -      -      -      -
  nvme-Samsung_SSD_970_EVO_Plus_500GB_S4EVNF0N420456K-part2   363G  5.02G      6     32   340K  3.04M
-----------------------------------------------------------  -----  -----  -----  -----  -----  -----

iostat:

Linux 5.9.14-arch1-1 (exia)     30/12/20        _x86_64_        (16 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           5.72    5.29    4.44    4.28    0.00   80.26

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
dm-0              0.01         0.02         0.00         0.00        312          0          0
dm-1              0.69         0.18         2.72         0.00       2220      34220          0
loop0             0.01         0.03         0.00         0.00        396          0          0
nvme0n1          45.54       347.79      3534.13         0.00    4374640   44453508          0
sda               5.51       402.33        59.44         0.00    5060642     747597          0
sdb              33.43         9.44      4375.22         0.00     118757   55033056          0
sdc              31.93         8.84      4372.38         0.00     111137   54997356          0

Screenshot_20201230_203840

bud4 commented 3 years ago

same problem with 0.8.3-1ubuntu12.5 on 5.4.0-60-generic #67~18.04.1-Ubuntu `48576.460707] INFO: task txg_sync:9452 blocked for more than 120 seconds. [48576.460762] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [48576.460810] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [48576.460861] txg_sync D 0 9452 2 0x80004000 [48576.460865] Call Trace: [48576.460875] schedule+0x293/0x720 [48576.461028] ? zio_taskq_member.isra.8.constprop.11+0x70/0x70 [zfs] [48576.461033] schedule+0x33/0xa0 [48576.461036] schedule_timeout+0x15d/0x320 [48576.461040] ? next_timer_interrupt+0xe0/0xe0 [48576.461044] io_schedule_timeout+0x1e/0x50 [48576.461059] cv_timedwait_common+0x159/0x1a0 [spl] [48576.461065] ? wait_woken+0x80/0x80 [48576.461074] cv_timedwait_io+0x19/0x20 [spl] [48576.461198] zio_wait+0x125/0x230 [zfs] [48576.461301] dsl_pool_sync+0xb9/0x410 [zfs] [48576.461419] spa_sync_iterate_to_convergence+0xd5/0x1c0 [zfs] [48576.461537] spa_sync+0x327/0x5c0 [zfs] [48576.461659] txg_sync_thread+0x283/0x340 [zfs] [48576.461780] ? txg_dispatch_callbacks+0x100/0x100 [zfs] [48576.461793] thread_generic_wrapper+0x74/0x90 [spl] [48576.461799] kthread+0x121/0x140 [48576.461811] ? thread_exit+0x20/0x20 [spl] [48576.461815] ? kthread_park+0x90/0x90 [48576.461819] ret_from_fork+0x35/0x40 [48576.461828] INFO: task zfs:15439 blocked for more than 120 seconds. [48576.461870] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [48576.461918] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [48576.461968] zfs D 0 15439 15419 0x80004002 [48576.461971] Call Trace: [48576.461976] schedule+0x293/0x720 [48576.461980] schedule+0x33/0xa0 [48576.461984] io_schedule+0x16/0x40 [48576.461992] cv_wait_common+0xdb/0x160 [spl] [48576.461997] ? wait_woken+0x80/0x80 [48576.462005] cv_wait_io+0x18/0x20 [spl] [48576.462127] txg_wait_synced_impl+0xd0/0x120 [zfs] [48576.462248] txg_wait_synced+0x10/0x40 [zfs] [48576.462367] zil_close+0x1a9/0x200 [zfs] [48576.462489] zfsvfs_teardown+0x306/0x320 [zfs] [48576.462610] zfs_suspend_fs+0x10/0x20 [zfs] [48576.462732] zfs_ioc_recv_impl+0x61f/0xa60 [zfs] [48576.462857] zfs_ioc_recv+0x1ed/0x330 [zfs] [48576.462865] ? ns_capable_common+0x2b/0x50 [48576.462868] ? capable+0x19/0x20 [48576.462977] ? priv_policy_ns.isra.1+0x4c/0x60 [zfs] [48576.463090] ? priv_policy.isra.2+0x10/0x20 [zfs] [48576.463095] ? _cond_resched+0x19/0x40 [48576.463099] ? kmalloc+0x199/0x280 [48576.463228] zfsdev_ioctl+0x584/0x620 [zfs] [48576.463232] do_vfs_ioctl+0xa9/0x640 [48576.463236] ? vfs_write+0x12e/0x1a0 [48576.463238] ksys_ioctl+0x75/0x80 [48576.463241] x64_sys_ioctl+0x1a/0x20 [48576.463247] do_syscall_64+0x57/0x190 [48576.463250] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [48576.463253] RIP: 0033:0x7f9b9b1bf317 [48576.463261] Code: Bad RIP value. [48576.463263] RSP: 002b:00007fffd987e1e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [48576.463266] RAX: ffffffffffffffda RBX: 000056053db731c0 RCX: 00007f9b9b1bf317 [48576.463267] RDX: 00007fffd987e380 RSI: 0000000000005a1b RDI: 0000000000000006 [48576.463269] RBP: 00007fffd987e380 R08: 0000000000000000 R09: 0000000000000000 [48576.463270] R10: 000056053d6ba010 R11: 0000000000000246 R12: 00007fffd988c528 [48576.463271] R13: 00007fffd9888670 R14: 00007fffd9886070 R15: 000056053d6d6cb8 [48697.292697] INFO: task txg_sync:9452 blocked for more than 241 seconds. [48697.292752] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [48697.292801] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [48697.292852] txg_sync D 0 9452 2 0x80004000 [48697.292855] Call Trace: [48697.292867] schedule+0x293/0x720 [48697.293020] ? zio_taskq_member.isra.8.constprop.11+0x70/0x70 [zfs] [48697.293025] schedule+0x33/0xa0 [48697.293027] schedule_timeout+0x15d/0x320 [48697.293032] ? next_timer_interrupt+0xe0/0xe0 [48697.293036] io_schedule_timeout+0x1e/0x50 [48697.293050] cv_timedwait_common+0x159/0x1a0 [spl] [48697.293056] ? wait_woken+0x80/0x80 [48697.293065] cv_timedwait_io+0x19/0x20 [spl] [48697.293191] zio_wait+0x125/0x230 [zfs] [48697.293293] dsl_pool_sync+0xb9/0x410 [zfs] [48697.293411] spa_sync_iterate_to_convergence+0xd5/0x1c0 [zfs] [48697.293527] spa_sync+0x327/0x5c0 [zfs] [48697.293649] txg_sync_thread+0x283/0x340 [zfs] [48697.293770] ? txg_dispatch_callbacks+0x100/0x100 [zfs] [48697.293783] thread_generic_wrapper+0x74/0x90 [spl] [48697.293790] kthread+0x121/0x140 [48697.293801] ? thread_exit+0x20/0x20 [spl] [48697.293805] ? kthread_park+0x90/0x90 [48697.293809] ret_from_fork+0x35/0x40 [48697.293818] INFO: task zfs:15439 blocked for more than 241 seconds. [48697.293861] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [48697.293909] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [48697.293959] zfs D 0 15439 15419 0x80004002 [48697.293961] Call Trace: [48697.293966] schedule+0x293/0x720 [48697.293971] schedule+0x33/0xa0 [48697.293974] io_schedule+0x16/0x40 [48697.293983] cv_wait_common+0xdb/0x160 [spl] [48697.293987] ? wait_woken+0x80/0x80 [48697.293995] cv_wait_io+0x18/0x20 [spl] [48697.294118] txg_wait_synced_impl+0xd0/0x120 [zfs] [48697.294239] txg_wait_synced+0x10/0x40 [zfs] [48697.294360] zil_close+0x1a9/0x200 [zfs] [48697.294482] zfsvfs_teardown+0x306/0x320 [zfs] [48697.294604] zfs_suspend_fs+0x10/0x20 [zfs] [48697.294727] zfs_ioc_recv_impl+0x61f/0xa60 [zfs] [48697.294853] zfs_ioc_recv+0x1ed/0x330 [zfs] [48697.294862] ? ns_capable_common+0x2b/0x50 [48697.294865] ? capable+0x19/0x20 [48697.294974] ? priv_policy_ns.isra.1+0x4c/0x60 [zfs] [48697.295083] ? priv_policy.isra.2+0x10/0x20 [zfs] [48697.295087] ? _cond_resched+0x19/0x40 [48697.295090] ? kmalloc+0x199/0x280 [48697.295214] zfsdev_ioctl+0x584/0x620 [zfs] [48697.295218] do_vfs_ioctl+0xa9/0x640 [48697.295222] ? vfs_write+0x12e/0x1a0 [48697.295224] ksys_ioctl+0x75/0x80 [48697.295227] x64_sys_ioctl+0x1a/0x20 [48697.295232] do_syscall_64+0x57/0x190 [48697.295235] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [48697.295238] RIP: 0033:0x7f9b9b1bf317 [48697.295246] Code: Bad RIP value. [48697.295248] RSP: 002b:00007fffd987e1e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [48697.295251] RAX: ffffffffffffffda RBX: 000056053db731c0 RCX: 00007f9b9b1bf317 [48697.295252] RDX: 00007fffd987e380 RSI: 0000000000005a1b RDI: 0000000000000006 [48697.295253] RBP: 00007fffd987e380 R08: 0000000000000000 R09: 0000000000000000 [48697.295255] R10: 000056053d6ba010 R11: 0000000000000246 R12: 00007fffd988c528 [48697.295256] R13: 00007fffd9888670 R14: 00007fffd9886070 R15: 000056053d6d6cb8 [69359.565083] INFO: task txg_sync:9452 blocked for more than 120 seconds. [69359.565139] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [69359.565187] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [69359.565238] txg_sync D 0 9452 2 0x80004000 [69359.565241] Call Trace: [69359.565252] schedule+0x293/0x720 [69359.565407] ? zio_taskq_member.isra.8.constprop.11+0x70/0x70 [zfs] [69359.565411] schedule+0x33/0xa0 [69359.565414] schedule_timeout+0x15d/0x320 [69359.565418] ? next_timer_interrupt+0xe0/0xe0 [69359.565422] io_schedule_timeout+0x1e/0x50 [69359.565437] cv_timedwait_common+0x159/0x1a0 [spl] [69359.565452] ? wait_woken+0x80/0x80 [69359.565461] cv_timedwait_io+0x19/0x20 [spl] [69359.565585] zio_wait+0x125/0x230 [zfs] [69359.565687] dsl_pool_sync+0xb9/0x410 [zfs] [69359.565804] spa_sync_iterate_to_convergence+0xd5/0x1c0 [zfs] [69359.565919] spa_sync+0x327/0x5c0 [zfs] [69359.566041] txg_sync_thread+0x283/0x340 [zfs] [69359.566163] ? txg_dispatch_callbacks+0x100/0x100 [zfs] [69359.566175] thread_generic_wrapper+0x74/0x90 [spl] [69359.566182] kthread+0x121/0x140 [69359.566194] ? thread_exit+0x20/0x20 [spl] [69359.566197] ? kthread_park+0x90/0x90 [69359.566201] ret_from_fork+0x35/0x40 [69359.566230] INFO: task zfs:9256 blocked for more than 120 seconds. [69359.566272] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [69359.566320] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [69359.566370] zfs D 0 9256 9236 0x80004002 [69359.566372] Call Trace: [69359.566377] schedule+0x293/0x720 [69359.566381] schedule+0x33/0xa0 [69359.566385] io_schedule+0x16/0x40 [69359.566394] cv_wait_common+0xdb/0x160 [spl] [69359.566399] ? wait_woken+0x80/0x80 [69359.566407] cv_wait_io+0x18/0x20 [spl] [69359.566530] txg_wait_synced_impl+0xd0/0x120 [zfs] [69359.566651] txg_wait_synced+0x10/0x40 [zfs] [69359.566771] zil_close+0x1a9/0x200 [zfs] [69359.566894] zfsvfs_teardown+0x306/0x320 [zfs] [69359.567017] zfs_suspend_fs+0x10/0x20 [zfs] [69359.567146] zfs_ioc_recv_impl+0x61f/0xa60 [zfs] [69359.567278] zfs_ioc_recv+0x1ed/0x330 [zfs] [69359.567286] ? ns_capable_common+0x2b/0x50 [69359.567289] ? capable+0x19/0x20 [69359.567401] ? priv_policy_ns.isra.1+0x4c/0x60 [zfs] [69359.567515] ? priv_policy.isra.2+0x10/0x20 [zfs] [69359.567519] ? _cond_resched+0x19/0x40 [69359.567523] ? __kmalloc+0x199/0x280 [69359.567653] zfsdev_ioctl+0x584/0x620 [zfs] [69359.567658] do_vfs_ioctl+0xa9/0x640 [69359.567662] ? vfs_write+0x12e/0x1a0 [69359.567664] ksys_ioctl+0x75/0x80 [69359.567667] x64_sys_ioctl+0x1a/0x20 [69359.567672] do_syscall_64+0x57/0x190 [69359.567676] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [69359.567679] RIP: 0033:0x7f59d1ee2317 [69359.567686] Code: Bad RIP value. [69359.567688] RSP: 002b:00007ffe8b382d18 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [69359.567691] RAX: ffffffffffffffda RBX: 000055f6fe3ab8f0 RCX: 00007f59d1ee2317 [69359.567693] RDX: 00007ffe8b382eb0 RSI: 0000000000005a1b RDI: 0000000000000006 [69359.567694] RBP: 00007ffe8b382eb0 R08: 0000000000000000 R09: 0000000000000000 [69359.567695] R10: 000055f6fe3a8010 R11: 0000000000000246 R12: 00007ffe8b391058 [69359.567697] R13: 00007ffe8b38d1a0 R14: 00007ffe8b38aba0 R15: 000055f6fe3f0018 [76720.410257] perf: interrupt took too long (2525 > 2500), lowering kernel.perf_event_max_sample_rate to 79000 [79410.636710] perf: interrupt took too long (3160 > 3156), lowering kernel.perf_event_max_sample_rate to 63250 [86089.080897] perf: interrupt took too long (3962 > 3950), lowering kernel.perf_event_max_sample_rate to 50250 [101412.897346] perf: interrupt took too long (4961 > 4952), lowering kernel.perf_event_max_sample_rate to 40250 [183121.965206] perf: interrupt took too long (6228 > 6201), lowering kernel.perf_event_max_sample_rate to 32000 [660349.114725] INFO: task txg_sync:9452 blocked for more than 120 seconds. [660349.114780] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [660349.114829] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [660349.114880] txg_sync D 0 9452 2 0x80004000 [660349.114884] Call Trace: [660349.114895] schedule+0x293/0x720 [660349.115034] ? zio_taskq_member.isra.8.constprop.11+0x70/0x70 [zfs] [660349.115039] schedule+0x33/0xa0 [660349.115041] schedule_timeout+0x15d/0x320 [660349.115045] ? next_timer_interrupt+0xe0/0xe0 [660349.115049] io_schedule_timeout+0x1e/0x50 [660349.115061] cv_timedwait_common+0x159/0x1a0 [spl] [660349.115067] ? wait_woken+0x80/0x80 [660349.115074] cv_timedwait_io+0x19/0x20 [spl] [660349.115192] zio_wait+0x125/0x230 [zfs] [660349.115285] dsl_pool_sync+0xb9/0x410 [zfs] [660349.115393] spa_sync_iterate_to_convergence+0xd5/0x1c0 [zfs] [660349.115500] spa_sync+0x327/0x5c0 [zfs] [660349.115612] txg_sync_thread+0x283/0x340 [zfs] [660349.115723] ? txg_dispatch_callbacks+0x100/0x100 [zfs] [660349.115735] thread_generic_wrapper+0x74/0x90 [spl] [660349.115742] kthread+0x121/0x140 [660349.115752] ? thread_exit+0x20/0x20 [spl] [660349.115756] ? kthread_park+0x90/0x90 [660349.115760] ret_from_fork+0x35/0x40 [660711.610599] INFO: task txg_sync:9452 blocked for more than 120 seconds. [660711.610655] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [660711.610704] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [660711.610755] txg_sync D 0 9452 2 0x80004000 [660711.610759] Call Trace: [660711.610769] __schedule+0x293/0x720 [660711.610923] ? zio_taskq_member.isra.8.constprop.11+0x70/0x70 [zfs] [660711.610928] schedule+0x33/0xa0 [660711.610930] schedule_timeout+0x15d/0x320 [660711.610934] ? next_timer_interrupt+0xe0/0xe0 [660711.610938] io_schedule_timeout+0x1e/0x50 [660711.610953] cv_timedwait_common+0x159/0x1a0 [spl] [660711.610959] ? wait_woken+0x80/0x80 [660711.610967] cv_timedwait_io+0x19/0x20 [spl] [660711.611093] zio_wait+0x125/0x230 [zfs] [660711.611195] dsl_pool_sync+0xb9/0x410 [zfs] [660711.611314] spa_sync_iterate_to_convergence+0xd5/0x1c0 [zfs] [660711.611429] spa_sync+0x327/0x5c0 [zfs] [660711.611551] txg_sync_thread+0x283/0x340 [zfs] [660711.611672] ? txg_dispatch_callbacks+0x100/0x100 [zfs] [660711.611685] thread_generic_wrapper+0x74/0x90 [spl] [660711.611692] kthread+0x121/0x140 [660711.611703] ? thread_exit+0x20/0x20 [spl] [660711.611707] ? kthread_park+0x90/0x90 [660711.611711] ret_from_fork+0x35/0x40 [660711.611746] INFO: task zfs:3980 blocked for more than 120 seconds. [660711.611789] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [660711.611837] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [660711.611887] zfs D 0 3980 3960 0x80004002 [660711.611890] Call Trace: [660711.611895] schedule+0x293/0x720 [660711.611899] schedule+0x33/0xa0 [660711.611903] io_schedule+0x16/0x40 [660711.611912] cv_wait_common+0xdb/0x160 [spl] [660711.611917] ? wait_woken+0x80/0x80 [660711.611925] cv_wait_io+0x18/0x20 [spl] [660711.612048] txg_wait_synced_impl+0xd0/0x120 [zfs] [660711.612173] txg_wait_synced+0x10/0x40 [zfs] [660711.612299] zil_close+0x1a9/0x200 [zfs] [660711.612428] zfsvfs_teardown+0x306/0x320 [zfs] [660711.612556] zfs_suspend_fs+0x10/0x20 [zfs] [660711.612684] zfs_ioc_recv_impl+0x61f/0xa60 [zfs] [660711.612815] zfs_ioc_recv+0x1ed/0x330 [zfs] [660711.612823] ? ns_capable_common+0x2b/0x50 [660711.612826] ? capable+0x19/0x20 [660711.612940] ? priv_policy_ns.isra.1+0x4c/0x60 [zfs] [660711.613052] ? priv_policy.isra.2+0x10/0x20 [zfs] [660711.613057] ? _cond_resched+0x19/0x40 [660711.613060] ? __kmalloc+0x199/0x280 [660711.613190] zfsdev_ioctl+0x584/0x620 [zfs] [660711.613194] do_vfs_ioctl+0xa9/0x640 [660711.613198] ? vfs_write+0x12e/0x1a0 [660711.613200] ksys_ioctl+0x75/0x80 [660711.613203] x64_sys_ioctl+0x1a/0x20 [660711.613208] do_syscall_64+0x57/0x190 [660711.613212] entry_SYSCALL_64_after_hwframe+0x44/0xa9 [660711.613215] RIP: 0033:0x7f4bc3e58317 [660711.613223] Code: Bad RIP value. [660711.613225] RSP: 002b:00007ffcb28f3c78 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 [660711.613228] RAX: ffffffffffffffda RBX: 000055c2798fb400 RCX: 00007f4bc3e58317 [660711.613229] RDX: 00007ffcb28f3e10 RSI: 0000000000005a1b RDI: 0000000000000006 [660711.613230] RBP: 00007ffcb28f3e10 R08: 0000000000000000 R09: 0000000000000000 [660711.613232] R10: 000055c2790a5010 R11: 0000000000000246 R12: 00007ffcb2901fb8 [660711.613233] R13: 00007ffcb28fe100 R14: 00007ffcb28fbb00 R15: 000055c279136578 [663611.579659] INFO: task txg_sync:9452 blocked for more than 120 seconds. [663611.579715] Tainted: P O 5.4.0-60-generic #67~18.04.1-Ubuntu [663611.579763] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [663611.579814] txg_sync D 0 9452 2 0x80004000 [663611.579818] Call Trace: [663611.579828] schedule+0x293/0x720 [663611.579965] ? zio_taskq_member.isra.8.constprop.11+0x70/0x70 [zfs] [663611.579970] schedule+0x33/0xa0 [663611.579972] schedule_timeout+0x15d/0x320 [663611.579976] ? next_timer_interrupt+0xe0/0xe0 [663611.579980] io_schedule_timeout+0x1e/0x50 [663611.579993] cv_timedwait_common+0x159/0x1a0 [spl] [663611.579998] ? wait_woken+0x80/0x80 [663611.580006] cv_timedwait_io+0x19/0x20 [spl] [663611.580121] zio_wait+0x125/0x230 [zfs] [663611.580211] dsl_pool_sync+0xb9/0x410 [zfs] [663611.580319] spa_sync_iterate_to_convergence+0xd5/0x1c0 [zfs] [663611.580424] spa_sync+0x327/0x5c0 [zfs] [663611.580535] txg_sync_thread+0x283/0x340 [zfs] [663611.580647] ? txg_dispatch_callbacks+0x100/0x100 [zfs] [663611.580659] thread_generic_wrapper+0x74/0x90 [spl] [663611.580665] kthread+0x121/0x140 [663611.580676] ? __thread_exit+0x20/0x20 [spl] [663611.580680] ? kthread_park+0x90/0x90 [663611.580684] ret_from_fork+0x35/0x40 zpool status pool: pool_z2_samba state: ONLINE scan: none requested config:

NAME                        STATE     READ WRITE CKSUM
pool_z2_samba               ONLINE       0     0     0
  raidz2-0                  ONLINE       0     0     0
    wwn-0x5000c500a7c07f63  ONLINE       0     0     0
    wwn-0x5000c500ad83729b  ONLINE       0     0     0
    wwn-0x5000c500a7dbd1bb  ONLINE       0     0     0
    wwn-0x5000c500a7db9e2f  ONLINE       0     0     0
    wwn-0x5000c500a7db9d8b  ONLINE       0     0     0
    wwn-0x5000c500a7cea9df  ONLINE       0     0     0
  raidz2-1                  ONLINE       0     0     0
    wwn-0x5000c500a7bfff6b  ONLINE       0     0     0
    wwn-0x5000c500a7ce8693  ONLINE       0     0     0
    wwn-0x5000c500a7eef78b  ONLINE       0     0     0
    wwn-0x5000c500a7eeb35b  ONLINE       0     0     0
    wwn-0x5000c500a7c6b46f  ONLINE       0     0     0
    wwn-0x5000c500a7bfe307  ONLINE       0     0     0
  raidz2-2                  ONLINE       0     0     0
    wwn-0x5000c500a7d01c8f  ONLINE       0     0     0
    wwn-0x5000c500a7d02c33  ONLINE       0     0     0
    wwn-0x5000c500ad9bb5cf  ONLINE       0     0     0
    wwn-0x5000c500a7ce7c33  ONLINE       0     0     0
    wwn-0x5000c500a7e845ab  ONLINE       0     0     0
    wwn-0x5000c500a7ceaf8f  ONLINE       0     0     0
  raidz2-3                  ONLINE       0     0     0
    wwn-0x5000c500a7c00967  ONLINE       0     0     0
    wwn-0x5000c500a7bfffcb  ONLINE       0     0     0
    wwn-0x5000c500ad82cfaf  ONLINE       0     0     0
    wwn-0x5000c500a7dc50bb  ONLINE       0     0     0
    wwn-0x5000c500a7eee40f  ONLINE       0     0     0
    wwn-0x5000c500a7ce7aff  ONLINE       0     0     0
logs
  mirror-4                  ONLINE       0     0     0
    wwn-0x55cd2e414d8f03c5  ONLINE       0     0     0
    wwn-0x55cd2e414d8f12f9  ONLINE       0     0     0
cache
  sdf                       ONLINE       0     0     0
  sdg                       ONLINE       0     0     0

zpool get all NAME PROPERTY VALUE SOURCE pool_z2_samba size 174T - pool_z2_samba capacity 48% - pool_z2_samba altroot - default pool_z2_samba health ONLINE - pool_z2_samba guid 15112324204784897998 - pool_z2_samba version - default pool_z2_samba bootfs - default pool_z2_samba delegation on default pool_z2_samba autoreplace off default pool_z2_samba cachefile - default pool_z2_samba failmode wait default pool_z2_samba listsnapshots off default pool_z2_samba autoexpand off default pool_z2_samba dedupditto 0 default pool_z2_samba dedupratio 1.35x - pool_z2_samba free 89,2T - pool_z2_samba allocated 84,8T - pool_z2_samba readonly off - pool_z2_samba ashift 0 default pool_z2_samba comment - default pool_z2_samba expandsize - - pool_z2_samba freeing 0 - pool_z2_samba fragmentation 22% - pool_z2_samba leaked 0 - pool_z2_samba multihost off default pool_z2_samba feature@async_destroy enabled local pool_z2_samba feature@empty_bpobj active local pool_z2_samba feature@lz4_compress active local pool_z2_samba feature@multi_vdev_crash_dump enabled local pool_z2_samba feature@spacemap_histogram active local pool_z2_samba feature@enabled_txg active local pool_z2_samba feature@hole_birth active local pool_z2_samba feature@extensible_dataset active local pool_z2_samba feature@embedded_data active local pool_z2_samba feature@bookmarks enabled local pool_z2_samba feature@filesystem_limits enabled local pool_z2_samba feature@large_blocks enabled local pool_z2_samba feature@large_dnode enabled local pool_z2_samba feature@sha512 enabled local pool_z2_samba feature@skein enabled local pool_z2_samba feature@edonr enabled local pool_z2_samba feature@userobj_accounting active local db -DD pool_z2_samba DDT-sha256-zap-duplicate: 82962123 entries, size 1219 on disk, 197 in core DDT-sha256-zap-unique: 468774376 entries, size 1072 on disk, 173 in core

DDT histogram (aggregated over all DDTs):

bucket allocated referenced

refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE

 1     447M   54.1T   46.0T   46.5T     447M   54.1T   46.0T   46.5T
 2    64.2M   7.80T   6.48T   6.55T     144M   17.5T   14.6T   14.8T
 4    10.9M   1.30T   1.10T   1.11T    51.5M   6.13T   5.18T   5.24T
 8    2.72M    328G    276G    280G    28.2M   3.33T   2.78T   2.83T
16     887K    107G   87.5G   88.7G    18.2M   2.20T   1.79T   1.82T
32     320K   38.7G   33.5G   33.9G    13.1M   1.58T   1.36T   1.38T
64    74.8K   8.95G   7.53G   7.64G    5.96M    728G    607G    616G

128 37.1K 4.45G 4.10G 4.12G 6.02M 737G 678G 682G 256 3.39K 338M 238M 247M 1.09M 108G 74.2G 77.3G 512 1.50K 169M 121M 124M 1.23M 142G 106G 108G 1K 520 56.1M 38.0M 39.3M 768K 83.8G 59.6G 61.3G 2K 159 15.8M 5.63M 6.26M 420K 41.8G 14.5G 16.1G 4K 84 9.38M 5.09M 5.34M 414K 45.4G 22.6G 23.9G 8K 15 1.26M 46K 120K 164K 14.4G 532M 1.28G 16K 1 128K 4K 7.99K 16.1K 2.02G 64.5M 129M 32K 2 256K 8K 16.0K 94.9K 11.9G 380M 758M 128K 1 128K 4K 7.99K 140K 17.5G 561M 1.10G Total 526M 63.7T 54.0T 54.5T 719M 86.8T 73.2T 74.1T

dedup = 1.36, compress = 1.18, copies = 1.01, dedup * compress / copies = 1.59

`

stale[bot] commented 2 years ago

This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.