Open lightrush opened 4 years ago
cfcs
commented
4 years ago This is on a raspberry pi with spinning disks connected by USB?
I've experienced very similar failures that turned out to be related to the RPI not being able to provide enough power to the USB ports - may be worth trying to generate disk i/o with dd or shred or ext4 or whatever to see if it's related to ZFS at all.
EDIT: That being said I guess it would be nice if ZFS would fail the vdev instead of hanging though.
lightrush
commented
4 years ago Pi 4, yes. WD Elements are externally powered disks so power isn't an issue. I've also ran ATA secure erase on these disks on this same Pi which walks the whole disks without an issue. I'm running a very similar setup on another Pi 4 with Ext4 on LVMRAID mirror. It's been running 24/7 since last July without a squeak. That's the source of the data set I'm trying to copy.
lightrush
commented
4 years ago Currently testing latest master.
lightrush
commented
4 years ago Latest master managed to complete the Syncthing data transfer. Good.
Then I did rm -rf the whole dir and that reproduced the hang. Bad.
[160106.267704] INFO: task z_wr_int:3549 blocked for more than 120 seconds.
[160106.274547] Tainted: P C OE 5.4.0-1012-raspi #12-Ubuntu
[160106.281584] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[160106.289674] z_wr_int D 0 3549 2 0x00000028
[160106.289690] Call trace:
[160106.289718] __switch_to+0x104/0x170
[160106.289733] __schedule+0x30c/0x7c0
[160106.289744] schedule+0x3c/0xb8
[160106.289755] io_schedule+0x20/0x40
[160106.289768] rq_qos_wait+0x100/0x178
[160106.289780] wbt_wait+0xb4/0xf0
[160106.289791] __rq_qos_throttle+0x38/0x50
[160106.289806] blk_mq_make_request+0x128/0x610
[160106.289821] generic_make_request+0xb4/0x2d8
[160106.289831] submit_bio+0x48/0x218
[160106.290106] vdev_disk_io_start+0x540/0x900 [zfs]
[160106.290363] zio_vdev_io_start+0xdc/0x2b8 [zfs]
[160106.290644] zio_nowait+0xd4/0x170 [zfs]
[160106.290908] vdev_queue_io_done+0x1ec/0x2a0 [zfs]
[160106.291155] zio_vdev_io_done+0xec/0x220 [zfs]
[160106.291407] zio_execute+0xac/0x108 [zfs]
[160106.291445] taskq_thread+0x304/0x580 [spl]
[160106.291463] kthread+0xfc/0x128
[160106.291475] ret_from_fork+0x10/0x1c
[160106.291488] INFO: task z_wr_int:3552 blocked for more than 120 seconds.
[160106.298342] Tainted: P C OE 5.4.0-1012-raspi #12-Ubuntu
[160106.305351] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[160106.313419] z_wr_int D 0 3552 2 0x00000028
[160106.313432] Call trace:
[160106.313447] __switch_to+0x104/0x170
[160106.313461] __schedule+0x30c/0x7c0
[160106.313471] schedule+0x3c/0xb8
[160106.313482] io_schedule+0x20/0x40
[160106.313495] rq_qos_wait+0x100/0x178
[160106.313506] wbt_wait+0xb4/0xf0
[160106.313517] __rq_qos_throttle+0x38/0x50
[160106.313529] blk_mq_make_request+0x128/0x610
[160106.313541] generic_make_request+0xb4/0x2d8
[160106.313552] submit_bio+0x48/0x218
[160106.313801] vdev_disk_io_start+0x540/0x900 [zfs]
[160106.314040] zio_vdev_io_start+0xdc/0x2b8 [zfs]
[160106.314279] zio_nowait+0xd4/0x170 [zfs]
[160106.314518] vdev_queue_io_done+0x1ec/0x2a0 [zfs]
[160106.314757] zio_vdev_io_done+0xec/0x220 [zfs]
[160106.314995] zio_execute+0xac/0x108 [zfs]
[160106.315031] taskq_thread+0x304/0x580 [spl]
[160106.315044] kthread+0xfc/0x128
[160106.315055] ret_from_fork+0x10/0x1c
[160106.315077] INFO: task txg_sync:3781 blocked for more than 120 seconds.
[160106.321914] Tainted: P C OE 5.4.0-1012-raspi #12-Ubuntu
[160106.328919] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[160106.336988] txg_sync D 0 3781 2 0x00000028
[160106.337004] Call trace:
[160106.337016] __switch_to+0x104/0x170
[160106.337029] __schedule+0x30c/0x7c0
[160106.337039] schedule+0x3c/0xb8
[160106.337048] schedule_timeout+0x9c/0x190
[160106.337059] io_schedule_timeout+0x28/0x48
[160106.337094] __cv_timedwait_common+0x1ac/0x1f8 [spl]
[160106.337127] __cv_timedwait_io+0x3c/0x50 [spl]
[160106.337372] zio_wait+0x138/0x2b8 [zfs]
[160106.337610] dsl_pool_sync+0x3fc/0x498 [zfs]
[160106.337847] spa_sync+0x530/0xeb8 [zfs]
[160106.338085] txg_sync_thread+0x2d8/0x460 [zfs]
[160106.338120] thread_generic_wrapper+0x74/0xa0 [spl]
[160106.338132] kthread+0xfc/0x128
[160106.338143] ret_from_fork+0x10/0x1c
[160106.338167] INFO: task rm:1342914 blocked for more than 120 seconds.
[160106.344739] Tainted: P C OE 5.4.0-1012-raspi #12-Ubuntu
[160106.351745] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[160106.359811] rm D 0 1342914 1342913 0x00000000
[160106.359824] Call trace:
[160106.359836] __switch_to+0x104/0x170
[160106.359848] __schedule+0x30c/0x7c0
[160106.359858] schedule+0x3c/0xb8
[160106.359869] io_schedule+0x20/0x40
[160106.359903] cv_wait_common+0x104/0x1b0 [spl]
[160106.359936] __cv_wait_io+0x30/0x40 [spl]
[160106.360179] txg_wait_open+0xa0/0x118 [zfs]
[160106.360416] dmu_free_long_range+0x414/0x4f0 [zfs]
[160106.360656] zfs_rmnode+0x2e8/0x3e0 [zfs]
[160106.360896] zfs_zinactive+0x12c/0x148 [zfs]
[160106.361135] zfs_inactive+0x78/0x1f8 [zfs]
[160106.361374] zpl_evict_inode+0x4c/0x70 [zfs]
[160106.361386] evict+0xcc/0x1c8
[160106.361396] iput+0x158/0x250
[160106.361407] do_unlinkat+0x1bc/0x2a0
[160106.361418] __arm64_sys_unlinkat+0x44/0x70
[160106.361427] el0_svc_common.constprop.0+0xe0/0x1e8
[160106.361435] el0_svc_handler+0x34/0xa0
[160106.361445] el0_svc+0x10/0x14Tested on another Pi 4 board to eliminate a potential hardware problem like bad memory, reproduced.
lightrush
commented
4 years ago I'm retrying the same transfer on the same hardware with the same software but using Ext4 on LUKS on LVM RAID1 instead of ZFS.
lightrush
commented
4 years ago Both data replication via Syncthing and rm-ing it completed twice on the same hardware using Ext4 on LUKS on LVM on mdraid mirror.
agiz10
commented
4 years ago | Type | Version/Name |
|---|---|
| Distribution Name | Ubuntu |
| Distribution Version | 20.04.1 |
| Linux Kernel | Linux 5.4.0-1019-raspi aarch64 |
| Architecture | arm64 |
| ZFS Version | 0.8.3-1ubuntu12.4 |
| SPL Version | 0.8.3-1ubuntu12.4 |
| Hardware | RaspberryPi 4b 4GB |
| System disk | USB SSD - ext4 - connected to USB2 |
| Storage Disks | 2x 14TB USB WD MyBook in ZFS mirror- connected to 2xUSB3 |
It seems i have identical errors as op. Random IO freezes that only resolve by rebooting the system. I first tried with single drive, and results were the same. I observed the freezes only when doing disk-intensive operations.
This freeze is a a bit different then the rest because i was running scrub and before scrub finished i only later noticed txg_sync were present in the logs (logs are below).
But scrub still shows as finished successfully. And the pool was responding as well. Then i did a rm -R * and a lot of the files got deleted untill the command froze (dreaded class=deadman in logs).
I can provide whatever logs you need or test something to help identify the cause of this problem and squish this bug. The problem is reproducable. I do not know where to go from here.
I get freezes in three scenarios that i observed so far:
rm -rfFrom the freeze today:
pool: uranpool
state: ONLINE
scan: resilvered 8.99T in 1 days 07:54:43 with 0 errors on Fri Oct 16 06:31:13 2020
remove: Removal of vdev 1 copied 32K in 0h0m, completed on Wed Oct 14 22:33:56 2020
96 memory used for removed device mappings
config:
NAME STATE READ WRITE CKSUM
uranpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
usb-WD_My_Book_25EE_1111111111111154-0:0 ONLINE 0 0 0
usb-WD_My_Book_25EE_2222222222222243-0:0 ONLINE 0 0 0
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c 19:47:46 0 0 0 0 0 0 0 0 0 1.6G 1.7G
total used free shared buff/cache available Mem: 3792 2477 895 4 419 1255 Swap: 6143 4 6139
capacity operations bandwidth
pool alloc free read write read write
-------------------------------------------- ----- ----- ----- ----- ----- -----
uranpool 9.03T 3.69T 149 136 57.5M 58.6M
mirror 9.03T 3.69T 150 137 57.9M 59.1M
usb-WD_My_Book_25EE_1111111111111154-0:0 - - 147 14 57.5M 598K
usb-WD_My_Book_25EE_2222222222222243-0:0 - - 1 123 9.20K 58.5M
-------------------------------------------- ----- ----- ----- ----- ----- -----
Total DISK READ: 0.00 B/s | Total DISK WRITE: 49.51 K/s
Current DISK READ: 0.00 B/s | Current DISK WRITE: 17.68 K/s
TID PRIO USER DISK READ DISK WRITE SWAPIN IO> COMMAND
463673 ?sys user 0.00 B 0.00 B 0.00 % 99.99 % smbd --foreground --no-process-group
2093 ?sys root 0.00 B 0.00 B 0.00 % 99.99 % [txg_sync]
Oct 15 10:20:09 uran kernel: INFO: task txg_sync:2094 blocked for more than 120 seconds. Oct 15 10:20:09 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:20:09 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:20:09 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:20:09 uran kernel: Call trace: Oct 15 10:20:09 uran kernel: __switch_to+0x104/0x170 Oct 15 10:20:09 uran kernel: __schedule+0x314/0x810 Oct 15 10:20:09 uran kernel: schedule+0x48/0xe8 Oct 15 10:20:09 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:20:09 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:20:09 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:20:09 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:20:09 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:20:09 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:20:09 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:20:09 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:20:09 uran kernel: kthread+0x104/0x130 Oct 15 10:20:09 uran kernel: ret_from_fork+0x10/0x1c Oct 15 10:21:38 uran PackageKit[3376127]: daemon quit Oct 15 10:21:38 uran systemd[1]: packagekit.service: Succeeded. Oct 15 10:22:10 uran kernel: INFO: task txg_sync:2094 blocked for more than 241 seconds. Oct 15 10:22:10 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:22:10 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:22:10 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:22:10 uran kernel: Call trace: Oct 15 10:22:10 uran kernel: __switch_to+0x104/0x170 Oct 15 10:22:10 uran kernel: __schedule+0x314/0x810 Oct 15 10:22:10 uran kernel: schedule+0x48/0xe8 Oct 15 10:22:10 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:22:10 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:22:10 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:22:10 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:22:10 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:22:10 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:22:10 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:22:10 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:22:10 uran kernel: kthread+0x104/0x130 Oct 15 10:22:10 uran kernel: ret_from_fork+0x10/0x1c Oct 15 10:24:10 uran kernel: INFO: task txg_sync:2094 blocked for more than 362 seconds. Oct 15 10:24:10 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:24:10 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:24:10 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:24:10 uran kernel: Call trace: Oct 15 10:24:10 uran kernel: __switch_to+0x104/0x170 Oct 15 10:24:10 uran kernel: __schedule+0x314/0x810 Oct 15 10:24:10 uran kernel: schedule+0x48/0xe8 Oct 15 10:24:10 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:24:10 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:24:10 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:24:10 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:24:10 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:24:10 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:24:10 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:24:10 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:24:10 uran kernel: kthread+0x104/0x130 Oct 15 10:24:10 uran kernel: ret_from_fork+0x10/0x1c Oct 15 10:25:01 uran CRON[3383262]: pam_unix(cron:session): session opened for user root by (uid=0) Oct 15 10:25:01 uran CRON[3383263]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Oct 15 10:25:01 uran CRON[3383262]: pam_unix(cron:session): session closed for user root Oct 15 10:26:11 uran kernel: INFO: task txg_sync:2094 blocked for more than 483 seconds. Oct 15 10:26:11 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:26:11 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:26:11 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:26:11 uran kernel: Call trace: Oct 15 10:26:11 uran kernel: __switch_to+0x104/0x170 Oct 15 10:26:11 uran kernel: __schedule+0x314/0x810 Oct 15 10:26:11 uran kernel: schedule+0x48/0xe8 Oct 15 10:26:11 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:26:11 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:26:11 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:26:11 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:26:11 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:26:11 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:26:11 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:26:11 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:26:11 uran kernel: kthread+0x104/0x130 Oct 15 10:26:11 uran kernel: ret_from_fork+0x10/0x1c Oct 15 10:28:12 uran kernel: INFO: task txg_sync:2094 blocked for more than 604 seconds. Oct 15 10:28:12 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:28:12 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:28:12 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:28:12 uran kernel: Call trace: Oct 15 10:28:12 uran kernel: __switch_to+0x104/0x170 Oct 15 10:28:12 uran kernel: __schedule+0x314/0x810 Oct 15 10:28:12 uran kernel: schedule+0x48/0xe8 Oct 15 10:28:12 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:28:12 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:28:12 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:28:12 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:28:12 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:28:12 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:28:12 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:28:12 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:28:12 uran kernel: kthread+0x104/0x130 Oct 15 10:28:12 uran kernel: ret_from_fork+0x10/0x1c ... Oct 15 10:30:13 uran kernel: INFO: task txg_sync:2094 blocked for more than 724 seconds. Oct 15 10:30:13 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:30:13 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:30:13 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:30:13 uran kernel: Call trace: Oct 15 10:30:13 uran kernel: __switch_to+0x104/0x170 Oct 15 10:30:13 uran kernel: __schedule+0x314/0x810 Oct 15 10:30:13 uran kernel: schedule+0x48/0xe8 Oct 15 10:30:13 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:30:13 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:30:13 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:30:13 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:30:13 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:30:13 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:30:13 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:30:13 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:30:13 uran kernel: kthread+0x104/0x130 Oct 15 10:30:13 uran kernel: ret_from_fork+0x10/0x1c Oct 15 10:32:14 uran kernel: INFO: task txg_sync:2094 blocked for more than 845 seconds. Oct 15 10:32:14 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:32:14 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:32:14 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:32:14 uran kernel: Call trace: Oct 15 10:32:14 uran kernel: __switch_to+0x104/0x170 Oct 15 10:32:14 uran kernel: __schedule+0x314/0x810 Oct 15 10:32:14 uran kernel: schedule+0x48/0xe8 Oct 15 10:32:14 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:32:14 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:32:14 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:32:14 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:32:14 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:32:14 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:32:14 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:32:14 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:32:14 uran kernel: kthread+0x104/0x130 Oct 15 10:32:14 uran kernel: ret_from_fork+0x10/0x1c Oct 15 10:34:15 uran kernel: INFO: task txg_sync:2094 blocked for more than 966 seconds. Oct 15 10:34:15 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:34:15 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:34:15 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:34:15 uran kernel: Call trace: Oct 15 10:34:15 uran kernel: __switch_to+0x104/0x170 Oct 15 10:34:15 uran kernel: __schedule+0x314/0x810 Oct 15 10:34:15 uran kernel: schedule+0x48/0xe8 Oct 15 10:34:15 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:34:15 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:34:15 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:34:15 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:34:15 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:34:15 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:34:15 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:34:15 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:34:15 uran kernel: kthread+0x104/0x130 Oct 15 10:34:15 uran kernel: ret_from_fork+0x10/0x1c Oct 15 10:35:01 uran CRON[3386234]: pam_unix(cron:session): session opened for user root by (uid=0) Oct 15 10:35:01 uran CRON[3386235]: (root) CMD (command -v debian-sa1 > /dev/null && debian-sa1 1 1) Oct 15 10:35:01 uran CRON[3386234]: pam_unix(cron:session): session closed for user root Oct 15 10:36:15 uran kernel: INFO: task txg_sync:2094 blocked for more than 1087 seconds. Oct 15 10:36:15 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:36:15 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:36:15 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:36:15 uran kernel: Call trace: Oct 15 10:36:15 uran kernel: __switch_to+0x104/0x170 Oct 15 10:36:15 uran kernel: __schedule+0x314/0x810 Oct 15 10:36:15 uran kernel: schedule+0x48/0xe8 Oct 15 10:36:15 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:36:15 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:36:15 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:36:15 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:36:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:36:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:36:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:36:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:36:15 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:36:16 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:36:16 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:36:16 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:36:16 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:36:16 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:36:16 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:36:16 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:36:16 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:36:16 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:36:16 uran kernel: kthread+0x104/0x130 Oct 15 10:36:16 uran kernel: ret_from_fork+0x10/0x1c Oct 15 10:38:16 uran kernel: INFO: task txg_sync:2094 blocked for more than 1208 seconds. Oct 15 10:38:16 uran kernel: Tainted: P C OE 5.4.0-1019-raspi #21-Ubuntu Oct 15 10:38:16 uran kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 15 10:38:16 uran kernel: txg_sync D 0 2094 2 0x00000028 Oct 15 10:38:16 uran kernel: Call trace: Oct 15 10:38:16 uran kernel: __switch_to+0x104/0x170 Oct 15 10:38:16 uran kernel: __schedule+0x314/0x810 Oct 15 10:38:16 uran kernel: schedule+0x48/0xe8 Oct 15 10:38:16 uran kernel: cv_wait_common+0x12c/0x170 [spl] Oct 15 10:38:16 uran kernel: __cv_wait+0x30/0x40 [spl] Oct 15 10:38:16 uran kernel: arc_read+0x160/0xdc8 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visitbp+0x37c/0x900 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visitbp+0x218/0x900 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visitbp+0x5ec/0x900 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visit_rootbp+0xcc/0x110 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visitds+0x140/0x460 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_visit+0xb4/0x290 [zfs] Oct 15 10:38:16 uran kernel: dsl_scan_sync+0x3f8/0x7d8 [zfs] Oct 15 10:38:16 uran kernel: spa_sync_iterate_to_convergence+0x124/0x1e8 [zfs] Oct 15 10:38:16 uran kernel: spa_sync+0x2ec/0x520 [zfs] Oct 15 10:38:16 uran kernel: txg_sync_thread+0x244/0x2a0 [zfs] Oct 15 10:38:16 uran kernel: thread_generic_wrapper+0x74/0xa0 [spl] Oct 15 10:38:16 uran kernel: kthread+0x104/0x130 Oct 15 10:38:16 uran kernel: ret_from_fork+0x10/0x1c ... Oct 16 06:31:14 uran zed[1687034]: eid=57 class=history_event pool_guid=0x5E9821C46223F9A5 Oct 16 06:31:15 uran zed[1687036]: eid=58 class=resilver_finish pool_guid=0x5E9821C46223F9A5 Oct 16 06:31:16 uran zed[1687043]: error: resilver_finish-notify.sh: eid=58: "mail" not installed ... Oct 16 12:59:08 uran zed[2763536]: eid=59 class=deadman pool_guid=0x5E9821C46223F9A5 vdev_path=/dev/disk/by-id/usb-WD_My_Book_25EE_2222222222222243-0:0-part1 Oct 16 12:59:09 uran zed[2763538]: eid=60 class=deadman pool_guid=0x5E9821C46223F9A5 vdev_path=/dev/disk/by-id/usb-WD_My_Book_25EE_2222222222222243-0:0-part1
11 1 0x01 12 3264 20045022062 164369774375694 name type data dmu_tx_assigned 4 4978191 dmu_tx_delay 4 0 dmu_tx_error 4 0 dmu_tx_suspended 4 0 dmu_tx_group 4 0 dmu_tx_memory_reserve 4 0 dmu_tx_memory_reclaim 4 0 dmu_tx_dirty_throttle 4 0 dmu_tx_dirty_delay 4 28002 dmu_tx_dirty_over_max 4 0 dmu_tx_dirty_frees_delay 4 7803 dmu_tx_quota 4 0
12 1 0x01 98 26656 20053414543 164422259257589 name type data hits 4 54090208 misses 4 2775485 demand_data_hits 4 2 demand_data_misses 4 2 demand_metadata_hits 4 51159488 demand_metadata_misses 4 643447 prefetch_data_hits 4 0 prefetch_data_misses 4 0 prefetch_metadata_hits 4 2930718 prefetch_metadata_misses 4 2132036 mru_hits 4 21274026 mru_ghost_hits 4 7435 mfu_hits 4 30053299 mfu_ghost_hits 4 20430 deleted 4 2914903 mutex_miss 4 422 access_skip 4 307 evict_skip 4 14622752 evict_not_enough 4 18017 evict_l2_cached 4 0 evict_l2_eligible 4 73021932544 evict_l2_ineligible 4 198834327552 evict_l2_skip 4 0 hash_elements 4 122896 hash_elements_max 4 360413 hash_collisions 4 1166586 hash_chains 4 12288 hash_chain_max 4 8 p 4 1165417578 c 4 1781544448 c_min 4 124267008 c_max 4 1988272128 size 4 1724062016 compressed_size 4 1079760384 uncompressed_size 4 4306244608 overhead_size 4 207561728 hdr_size 4 41035192 data_size 4 743559168 metadata_size 4 543762944 dbuf_size 4 81275688 dnode_size 4 233893344 bonus_size 4 80535680 anon_size 4 9024000 anon_evictable_data 4 0 anon_evictable_metadata 4 0 mru_size 4 1155947520 mru_evictable_data 4 743415808 mru_evictable_metadata 4 278911488 mru_ghost_size 4 568823808 mru_ghost_evictable_data 4 372506624 mru_ghost_evictable_metadata 4 196317184 mfu_size 4 122350592 mfu_evictable_data 4 0 mfu_evictable_metadata 4 14849536 mfu_ghost_size 4 427843584 mfu_ghost_evictable_data 4 0 mfu_ghost_evictable_metadata 4 427843584 l2_hits 4 0 l2_misses 4 0 l2_feeds 4 0 l2_rw_clash 4 0 l2_read_bytes 4 0 l2_write_bytes 4 0 l2_writes_sent 4 0 l2_writes_done 4 0 l2_writes_error 4 0 l2_writes_lock_retry 4 0 l2_evict_lock_retry 4 0 l2_evict_reading 4 0 l2_evict_l1cached 4 0 l2_free_on_write 4 0 l2_abort_lowmem 4 0 l2_cksum_bad 4 0 l2_io_error 4 0 l2_size 4 0 l2_asize 4 0 l2_hdr_size 4 0 memory_throttle_count 4 0 memory_direct_count 4 928 memory_indirect_count 4 187112 memory_all_bytes 4 3976544256 memory_free_bytes 4 1212428288 memory_available_bytes 3 1150296064 arc_no_grow 4 0 arc_tempreserve 4 0 arc_loaned_bytes 4 0 arc_prune 4 35532 arc_meta_used 4 980502848 arc_meta_limit 4 1491204096 arc_dnode_limit 4 149120409 arc_meta_max 4 2016757248 arc_meta_min 4 16777216 async_upgrade_sync 4 1118 demand_hit_predictive_prefetch 4 164808 demand_hit_prescient_prefetch 4 3518483 arc_need_free 4 0 arc_sys_free 4 62133504 arc_raw_size 4 0
1602871076 zio.c:1919:zio_deadman_impl(): slow zio[5]: zio=ffff00007fa39d10 timestamp=139014768748992 delta=25433705563767 queued=0 io=0 path=/dev/disk/by-id/usb-WD_My_Book_25EE_2222222222222243-0:0-part1 last=139014860493029 type=2 priority=3 flags=0x180880 stage=0x100000 pipeline=0x1700000 pipeline-trace=0x100001 objset=0 object=0 level=0 blkid=41 offset=2105049370624 size=4096 error=0 1602871076 zio.c:1919:zio_deadman_impl(): slow zio[5]: zio=ffff00007fa3a6c0 timestamp=139014768739863 delta=25433705755729 queued=0 io=0 path=/dev/disk/by-id/usb-WD_My_Book_25EE_2222222222222243-0:0-part1 last=139014860493029 type=2 priority=3 flags=0x180880 stage=0x100000 pipeline=0x1700000 pipeline-trace=0x100001 objset=0 object=0 level=0 blkid=41 offset=7924248633344 size=4096 error=0 1602871076 zio.c:1919:zio_deadman_impl(): slow zio[5]: zio=ffff00006e5b3548 timestamp=139014769038474 delta=25433705642229 queued=0 io=0 path=/dev/disk/by-id/usb-WD_My_Book_25EE_2222222222222243-0:0-part1 last=139014860493029 type=2 priority=3 flags=0x180880 stage=0x100000 pipeline=0x1700000 pipeline-trace=0x100001 objset=0 object=0 level=0 blkid=2 offset=2105049366528 size=4096 error=0 1602871076 zio.c:1919:zio_deadman_impl(): slow zio[5]: zio=ffff00006e5b1360 timestamp=139014769026992 delta=25433705841137 queued=0 io=0 path=/dev/disk/by-id/usb-WD_My_Book_25EE_2222222222222243-0:0-part1 last=139014860493029 type=2 priority=3 flags=0x180880 stage=0x100000 pipeline=0x1700000 pipeline-trace=0x100001 objset=0 object=0 level=0 blkid=2 offset=7924248629248 size=4096 error=0 1602871076 zio.c:1919:zio_deadman_impl(): slow zio[5]: zio=ffff00006e5b1d10 timestamp=139014769012937 delta=25433706041247 queued=0 io=0 path=/dev/disk/by-id/usb-WD_My_Book_25EE_2222222222222243-0:0-part1 last=139014860493029 type=2 priority=3 flags=0x180880 stage=0x100000 pipeline=0x1700000 pipeline-trace=0x100001 objset=0 object=0 level=0 blkid=2 offset=2306569748480 size=4096 error=0 1602871076 zio.c:1967:zio_deadman(): zio_wait waiting for hung I/O to pool 'uranpool' 1602871080 spa_misc.c:605:spa_deadman(): slow spa_sync: started 25437 seconds ago, calls 424 1602871080 vdev.c:4670:vdev_deadman(): slow vdev: /dev/disk/by-id/usb-WD_My_Book_25EE_2222222222222243-0:0-part1 has 2 active IOs 1602871080 zio.c:1919:zio_deadman_impl(): slow zio[0]: zio=ffff0000f272e0e0 timestamp=139014769399714 delta=25437795829857 queued=0 io=139014770094326 path=/dev/disk/by-id/usb-WD_My_Book_25EE_2222222222222243-0:0-part1 last=139014860493029 type=2 priority=3 flags=0x40080c80 stage=0x100000 pipeline=0x1700000 pipeline-trace=0x100001 objset=0 object=0 level=0 blkid=0 offset=2306569768960 size=24576 error=0 1602871080 zio.c:1967:zio_deadman(): spa_deadman waiting for hung I/O to pool 'uranpool'
------------------------------------------------------------------------
ZFS Subsystem Report Fri Oct 16 19:59:37 2020
Linux 5.4.0-1019-raspi 0.8.3-1ubuntu12.4
Machine: uran (aarch64) 0.8.3-1ubuntu12.4
WARNING: Pages are deprecated, please use "--section"
ARC status: HEALTHY
Memory throttle count: 0
ARC size (current): 86.7 % 1.6 GiB
Target size (adaptive): 89.6 % 1.7 GiB
Min size (hard limit): 6.2 % 118.5 MiB
Max size (high water): 16:1 1.9 GiB
Most Frequently Used (MFU) cache size: 9.6 % 116.7 MiB
Most Recently Used (MRU) cache size: 90.4 % 1.1 GiB
Metadata cache size (hard limit): 75.0 % 1.4 GiB
Metadata cache size (current): 65.8 % 935.1 MiB
Dnode cache size (hard limit): 10.0 % 142.2 MiB
Dnode cache size (current): 156.8 % 223.1 MiB
ARC hash breakdown:
Elements max: 360.4k
Elements current: 34.1 % 122.9k
Collisions: 1.2M
Chain max: 8
Chains: 12.3k
ARC misc:
Deleted: 2.9M
Mutex misses: 422
Eviction skips: 14.6M
MemTotal: 3883344 kB MemFree: 929584 kB MemAvailable: 1299796 kB Buffers: 38140 kB Cached: 295272 kB SwapCached: 620 kB Active: 228036 kB Inactive: 157544 kB Active(anon): 19768 kB Inactive(anon): 42800 kB Active(file): 208268 kB Inactive(file): 114744 kB Unevictable: 17244 kB Mlocked: 17244 kB SwapTotal: 6291452 kB SwapFree: 6286580 kB Dirty: 0 kB Writeback: 0 kB AnonPages: 69140 kB Mapped: 55508 kB Shmem: 4104 kB KReclaimable: 97364 kB Slab: 1282460 kB SReclaimable: 97364 kB SUnreclaim: 1185096 kB KernelStack: 4032 kB PageTables: 2540 kB NFS_Unstable: 0 kB Bounce: 0 kB WritebackTmp: 0 kB CommitLimit: 8233124 kB Committed_AS: 285468 kB VmallocTotal: 135290159040 kB VmallocUsed: 145128 kB VmallocChunk: 0 kB Percpu: 2240 kB CmaTotal: 65536 kB CmaFree: 62656 kB
txg birth state ndirty nread nwritten reads writes otime qtime wtime stime ... 1270487 139013638627326 C 6889984 77824 3244032 19 237 578093481 7074 71130 286069963 1270488 139014216720807 S 0 0 0 0 0 525613741 5981 57056 0 1270489 139014742334548 W 0 0 0 0 0 505985648 8592 0 0 1270490 139015248320196 O 0 0 0 0 0 0 0 0 0
morkol-dot
commented
3 years ago I have the same issue (RPi 4b 8GB with a WD digital 2,5" external drive) and was also able to replicate it with a similar RPi 4 and a similar WD digital 2,5" external drive. Both with additional external power supplys (powered USB hub). Might be an issue that's caused by that specific disk / controller? Because I don't have the issue with a similar Seagate 2,5" drive.
stale[bot]
commented
2 years ago This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.
stale[bot]
commented
1 year ago This issue has been automatically marked as "stale" because it has not had any activity for a while. It will be closed in 90 days if no further activity occurs. Thank you for your contributions.
System information
Describe the problem you're observing
Tried to tranfer 3.4TB of data using Syncthing from another device on the network. Syncthing hung on IO after 150GB, 1.2TB, and 1.3TB on the 3 out of 3 attempts to do the transfer. I did scrub after the first hang. Then I decided to
rm -rfthe data and retry butrmhung in the same fashion. The disks are brand new WD Elements 8TB, non-SMR. They have been ATA secure erased and show no issues in SMART. The zpool contains a simple 2-way mirror. Load is 6.73, 6.67, 6.21 while ZFS is hung. The hang seems stay indefinitely until the machine is rebooted. I had it hung for over 12 hours on one occasion before rebooting. The system stays alive since root isn't on the zpool. There are no errors inzpool status. It looks very deadlock-y.Describe how to reproduce the problem
I don't know if you can reproduce it unless you have the same setup and data set, but this is very repeatable here so I can run debug versions and collect logs if that would help narrow the issue down. The system is disposable at the moment so running arbitrary versions of ZFS and reproducing it shouldn't be an issue.
Include any warning/errors/backtraces from the system logs
This is what I see in the log: