simultaneous zfs snapshot access hangs

[mailinglists35]

01:18:44 up 9:06, 3 users, load average: 8.83, 7.56, 6.23

this is an ubuntu 12.04 lts amd64 test system doing nothing (yet)

root@mailhost:~# dpkg -l|grep zfs ii libzfs1 0.6.0.67-0ubuntu1~precise1 Native ZFS filesystem library for Linux ii mountall 2.36-zfs1 filesystem mounting tool ii ubuntu-zfs 6~precise Native ZFS filesystem metapackage for Ubuntu. ii zfs-auto-snapshot 1.0.8-0ubuntu1~precise1 ZFS Automatic Snapshot Service ii zfs-dkms 0.6.0.67-0ubuntu1~precise1 Native ZFS filesystem kernel modules for Linux ii zfsutils 0.6.0.67-0ubuntu1~precise1 Native ZFS management utilities for Linux

[ 8.397519] SPL: Loaded module v0.6.0.67-rc9, using hostid 0xa8c04711

root@mailhost:~# cat /etc/apt/sources.list.d/zfs-native-daily-precise.list deb http://ppa.launchpad.net/zfs-native/daily/ubuntu precise main deb-src http://ppa.launchpad.net/zfs-native/daily/ubuntu precise main

Linux mailhost 3.2.0-27-generic #43-Ubuntu SMP Fri Jul 6 14:25:57 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

somehow cron managed to overlap two jobs that create a snapshot luckyly the zpool is on a dedicated pair of disks, and I have console acces to reset the box (reboot fails at "killing all remaining processes" then hangs at "unmounting local filesystems - or I don't have that much patience to see how long it waits. uhm, I'll just let it overnight to see if it eventually reboots, sleeping now.)

1224 ? Ss 0:00 cron 35459 ? S 0:00 CRON 35460 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 35461 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 35484 ? D 0:00 | zfs snapshot -o com.sun:auto-snapshot-desc - -r fulldisk@zfs-auto-snapfrequent-2012-07-28-0015 36090 ? S 0:00 CRON 36091 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 36092 ? S 0:00 | run-parts --report /etc/cron.hourly 36093 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=24 // 36095 ? D 0:00 | zpool status 36799 ? S 0:00 CRON 36800 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 36801 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 36803 ? D 0:00 | zpool status 36849 ? S 0:00 CRON 36850 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 36851 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 36853 ? D 0:00 | zpool status 38077 ? S 0:00 CRON 38078 ? Ss 0:00 /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 38079 ? S 0:00 /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 38081 ? D 0:00 zpool status

[mailinglists35]

don't know if this is related, but:

root@mailhost:/mnt/zfs/fulldisk/.zfs/snapshot# ls -lat ENTER

dmesg:

[ 23.558479] eth0: no IPv6 routers present [ 466.320772] VERIFY(!RW_LOCK_HELD(&l->l_rwlock)) failed [ 466.320856] SPLError: 338:0:(zap.c:444:zap_leaf_pageout()) SPL PANIC [ 466.320932] SPL: Showing stack for process 338 [ 466.320936] Pid: 338, comm: arc_adapt Tainted: P IO 3.2.0-27-generic #43-Ubuntu [ 466.320938] Call Trace: [ 466.320953] [] spl_debug_dumpstack+0x27/0x40 [spl] [ 466.320961] [] spl_debug_bug+0x82/0xe0 [spl] [ 466.321000] [] zap_leaf_pageout+0x73/0xc0 [zfs] [ 466.321019] [] dbuf_evict_user+0x49/0x80 [zfs] [ 466.321038] [] dbuf_clear+0x29/0x180 [zfs] [ 466.321056] [] dbuf_evict+0x16/0x30 [zfs] [ 466.321074] [] dbuf_do_evict+0x43/0x70 [zfs] [ 466.321090] [] arc_do_user_evicts+0x50/0xd0 [zfs] [ 466.321107] [] ? arc_adjust_meta+0x1e0/0x1e0 [zfs] [ 466.321123] [] arc_adapt_thread+0x7e/0xd0 [zfs] [ 466.321132] [] thread_generic_wrapper+0x78/0x90 [spl] [ 466.321139] [] ? __thread_create+0x310/0x310 [spl] [ 466.321145] [] kthread+0x8c/0xa0 [ 466.321150] [] kernel_thread_helper+0x4/0x10 [ 466.321154] [] ? flush_kthread_worker+0xa0/0xa0 [ 466.321158] [] ? gs_change+0x13/0x13 [ 466.321280] SPL: Dumping log to /tmp/spl-log.1343567577.338

/tmp/spl-log.1343567577.338 uploaded here: http://dl.transfer.ro/transfer_ro-29jul-eefa5255a5d0.zip

[mailinglists35]

root@mailhost:/var/tmp# dmesg | egrep 'spl|zfs' [261182.955186] [] ? kmem_free_debug+0x4b/0x150 [spl] [261182.955229] [] zap_get_leaf_byblk+0xe4/0x2d0 [zfs] [261182.955260] [] zap_deref_leaf+0x6d/0x80 [zfs] [261182.955290] [] fzap_add_cd+0x46/0x110 [zfs] [261182.955319] [] fzap_add+0x71/0x80 [zfs] [261182.955348] [] zap_add+0x11d/0x1a0 [zfs] [261182.955374] [] dsl_dataset_snapshot_sync+0x3a5/0x600 [zf ] [261182.955399] [] ? dsl_dir_space_available+0xd8/0x150 [zfs] [261182.955422] [] snapshot_sync+0x2e/0xd0 [zfs] [261182.955443] [] ? snapshot_check+0x2b/0xe0 [zfs] [261182.955470] [] dsl_sync_task_group_sync+0x123/0x210 [zfs] [261182.955495] [] dsl_pool_sync+0x1eb/0x450 [zfs] [261182.955523] [] spa_sync+0x38e/0xa00 [zfs] [261182.955553] [] txg_sync_thread+0x286/0x450 [zfs] [261182.955582] [] ? txg_init+0x250/0x250 [zfs] [261182.955591] [] thread_generic_wrapper+0x78/0x90 [spl] [261182.955599] [] ? thread_create+0x310/0x310 [spl] [261182.955626] INFO: task zfs:43990 blocked for more than 120 seconds. [261182.955688] zfs D ffffffff81806080 0 43990 43967 0x00000000 [261182.955720] [] cv_wait_common+0x98/0x190 [spl] [261182.955738] [] cv_wait+0x13/0x20 [spl] [261182.955767] [] txg_wait_synced+0xb3/0x190 [zfs] [261182.955794] [] dsl_sync_task_group_wait+0x17d/0x2a0 [zfs] [261182.955816] [] dmu_objset_snapshot+0x2c4/0x310 [zfs] [261182.955844] [] ? spa_open+0x13/0x20 [zfs] [261182.955874] [] zfs_ioc_snapshot+0x1c4/0x210 [zfs] [261182.955903] [] zfsdev_ioctl+0xdc/0x1b0 [zfs] [261302.928694] [] ? kmem_free_debug+0x4b/0x150 [spl] [261302.928736] [] zap_get_leaf_byblk+0xe4/0x2d0 [zfs] [261302.928767] [] zap_deref_leaf+0x6d/0x80 [zfs] [261302.928797] [] fzap_add_cd+0x46/0x110 [zfs] [261302.928825] [] fzap_add+0x71/0x80 [zfs] [261302.928854] [] zap_add+0x11d/0x1a0 [zfs] [261302.928879] [] dsl_dataset_snapshot_sync+0x3a5/0x600 [zf ] [261302.928905] [] ? dsl_dir_space_available+0xd8/0x150 [zfs] [261302.928927] [] snapshot_sync+0x2e/0xd0 [zfs] [261302.928948] [] ? snapshot_check+0x2b/0xe0 [zfs] [261302.928974] [] dsl_sync_task_group_sync+0x123/0x210 [zfs] [261302.929000] [] dsl_pool_sync+0x1eb/0x450 [zfs] [261302.929028] [] spa_sync+0x38e/0xa00 [zfs] [261302.929058] [] txg_sync_thread+0x286/0x450 [zfs] [261302.929087] [] ? txg_init+0x250/0x250 [zfs] [261302.929095] [] thread_generic_wrapper+0x78/0x90 [spl] [261302.929103] [] ? thread_create+0x310/0x310 [spl] [261302.929131] INFO: task zfs:43990 blocked for more than 120 seconds. [261302.929193] zfs D ffffffff81806080 0 43990 43967 0x00000000 [261302.929225] [] cv_wait_common+0x98/0x190 [spl] [261302.929242] [] cv_wait+0x13/0x20 [spl] [261302.929271] [] txg_wait_synced+0xb3/0x190 [zfs] [261302.929298] [] dsl_sync_task_group_wait+0x17d/0x2a0 [zfs] [261302.929320] [] dmu_objset_snapshot+0x2c4/0x310 [zfs] [261302.929349] [] ? spa_open+0x13/0x20 [zfs] [261302.929379] [] zfs_ioc_snapshot+0x1c4/0x210 [zfs] [261302.929408] [] zfsdev_ioctl+0xdc/0x1b0 [zfs] [261422.902212] [] ? kmem_free_debug+0x4b/0x150 [spl] [261422.902254] [] zap_get_leaf_byblk+0xe4/0x2d0 [zfs] [261422.902286] [] zap_deref_leaf+0x6d/0x80 [zfs] [261422.902315] [] fzap_add_cd+0x46/0x110 [zfs] [261422.902344] [] fzap_add+0x71/0x80 [zfs] [261422.902372] [] zap_add+0x11d/0x1a0 [zfs] [261422.902398] [] dsl_dataset_snapshot_sync+0x3a5/0x600 [zf ] [261422.902423] [] ? dsl_dir_space_available+0xd8/0x150 [zfs] [261422.902445] [] snapshot_sync+0x2e/0xd0 [zfs] [261422.902466] [] ? snapshot_check+0x2b/0xe0 [zfs] [261422.902493] [] dsl_sync_task_group_sync+0x123/0x210 [zfs] [261422.902518] [] dsl_pool_sync+0x1eb/0x450 [zfs] [261422.902547] [] spa_sync+0x38e/0xa00 [zfs] [261422.902576] [] txg_sync_thread+0x286/0x450 [zfs] [261422.902606] [] ? txg_init+0x250/0x250 [zfs] [261422.902615] [] thread_generic_wrapper+0x78/0x90 [spl] [261422.902622] [] ? thread_create+0x310/0x310 [spl] [261422.902649] INFO: task zfs:43990 blocked for more than 120 seconds. [261422.902711] zfs D ffffffff81806080 0 43990 43967 0x00000000 [261422.902743] [] cv_wait_common+0x98/0x190 [spl] [261422.902761] [] cv_wait+0x13/0x20 [spl] [261422.902790] [] txg_wait_synced+0xb3/0x190 [zfs] [261422.902817] [] dsl_sync_task_group_wait+0x17d/0x2a0 [zfs] [261422.902839] [] dmu_objset_snapshot+0x2c4/0x310 [zfs] [261422.902867] [] ? spa_open+0x13/0x20 [zfs] [261422.902897] [] zfs_ioc_snapshot+0x1c4/0x210 [zfs] [261422.902927] [] zfsdev_ioctl+0xdc/0x1b0 [zfs] [261542.875737] [] ? kmem_free_debug+0x4b/0x150 [spl] [261542.875779] [] zap_get_leaf_byblk+0xe4/0x2d0 [zfs] [261542.875810] [] zap_deref_leaf+0x6d/0x80 [zfs] [261542.875839] [] fzap_add_cd+0x46/0x110 [zfs] [261542.875868] [] fzap_add+0x71/0x80 [zfs] [261542.875896] [] zap_add+0x11d/0x1a0 [zfs] [261542.875922] [] dsl_dataset_snapshot_sync+0x3a5/0x600 [zf ] [261542.875947] [] ? dsl_dir_space_available+0xd8/0x150 [zfs] [261542.875969] [] snapshot_sync+0x2e/0xd0 [zfs] [261542.875991] [] ? snapshot_check+0x2b/0xe0 [zfs] [261542.876017] [] dsl_sync_task_group_sync+0x123/0x210 [zfs] [261542.876042] [] dsl_pool_sync+0x1eb/0x450 [zfs] [261542.876070] [] spa_sync+0x38e/0xa00 [zfs] [261542.876100] [] txg_sync_thread+0x286/0x450 [zfs] [261542.876129] [] ? txg_init+0x250/0x250 [zfs] [261542.876138] [] thread_generic_wrapper+0x78/0x90 [spl] [261542.876146] [] ? thread_create+0x310/0x310 [spl] [261542.876172] INFO: task zfs:43990 blocked for more than 120 seconds. [261542.876234] zfs D ffffffff81806080 0 43990 43967 0x00000000 [261542.876267] [] cv_wait_common+0x98/0x190 [spl] [261542.876284] [] cv_wait+0x13/0x20 [spl] [261542.876313] [] txg_wait_synced+0xb3/0x190 [zfs] [261542.876340] [] dsl_sync_task_group_wait+0x17d/0x2a0 [zfs] [261542.876363] [] dmu_objset_snapshot+0x2c4/0x310 [zfs] [261542.876391] [] ? spa_open+0x13/0x20 [zfs] [261542.876421] [] zfs_ioc_snapshot+0x1c4/0x210 [zfs] [261542.876450] [] zfsdev_ioctl+0xdc/0x1b0 [zfs] [261662.849258] [] ? kmem_free_debug+0x4b/0x150 [spl] [261662.849301] [] zap_get_leaf_byblk+0xe4/0x2d0 [zfs] [261662.849332] [] zap_deref_leaf+0x6d/0x80 [zfs] [261662.849362] [] fzap_add_cd+0x46/0x110 [zfs] [261662.849391] [] fzap_add+0x71/0x80 [zfs] [261662.849420] [] zap_add+0x11d/0x1a0 [zfs] [261662.849445] [] dsl_dataset_snapshot_sync+0x3a5/0x600 [zf ] [261662.849471] [] ? dsl_dir_space_available+0xd8/0x150 [zfs] [261662.849493] [] snapshot_sync+0x2e/0xd0 [zfs] [261662.849514] [] ? snapshot_check+0x2b/0xe0 [zfs] [261662.849540] [] dsl_sync_task_group_sync+0x123/0x210 [zfs] [261662.849566] [] dsl_pool_sync+0x1eb/0x450 [zfs] [261662.849594] [] spa_sync+0x38e/0xa00 [zfs] [261662.849623] [] txg_sync_thread+0x286/0x450 [zfs] [261662.849653] [] ? txg_init+0x250/0x250 [zfs] [261662.849661] [] thread_generic_wrapper+0x78/0x90 [spl] [261662.849669] [] ? __thread_create+0x310/0x310 [spl] [261662.849695] INFO: task zfs:43990 blocked for more than 120 seconds. [261662.849758] zfs D ffffffff81806080 0 43990 43967 0x00000000 [261662.849790] [] cv_wait_common+0x98/0x190 [spl] [261662.849808] [] __cv_wait+0x13/0x20 [spl] [261662.849837] [] txg_wait_synced+0xb3/0x190 [zfs] [261662.849864] [] dsl_sync_task_group_wait+0x17d/0x2a0 [zfs] [261662.849886] [] dmu_objset_snapshot+0x2c4/0x310 [zfs] [261662.849915] [] ? spa_open+0x13/0x20 [zfs] [261662.849944] [] zfs_ioc_snapshot+0x1c4/0x210 [zfs] [261662.849974] [] zfsdev_ioctl+0xdc/0x1b0 [zfs]

root@mailhost:/var/tmp# ps axfww|egrep -i 'cron|sync|zfs|zpool' 26 ? S 0:01 _ [syncsupers] 488 ? S< 0:00 [zfs_iputtaskq/] 490 ? D< 0:39 [txgsync] 32965 pts/3 S+ 0:00 egrep --color=auto -i cron|sync|zfs|zpoo 1204 ? Ss 0:00 cron 43965 ? S 0:00 CRON 43966 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 43967 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 43990 ? D 0:00 | zfs snapshot -o com.sun:auto-snapshot-desc - -r fulldisk@zfs-auto-snapfrequent-2012-08-02-0100 44115 ? S 0:00 CRON 44116 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44117 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44119 ? D 0:00 | zpool status 44127 ? S 0:00 CRON 44128 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 44129 ? S 0:00 | run-parts --report /etc/cron.hourly 44130 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 44132 ? D 0:00 | zpool status 44170 ? S 0:00 CRON 44171 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44172 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44174 ? D 0:00 | zpool status 44225 ? S 0:00 CRON 44226 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44227 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44229 ? D 0:00 | zpool status 44287 ? S 0:00 CRON 44288 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44289 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44291 ? D 0:00 | zpool status 44339 ? S 0:00 CRON 44340 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44341 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44343 ? D 0:00 | zpool status 44349 ? S 0:00 CRON 44350 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 44351 ? S 0:00 | run-parts --report /etc/cron.hourly 44352 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 44354 ? D 0:00 | zpool status 44400 ? S 0:00 CRON 44401 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44402 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44404 ? D 0:00 | zpool status 44474 ? S 0:00 CRON 44475 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44476 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44478 ? D 0:00 | zpool status 44517 ? S 0:00 CRON 44518 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44519 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44521 ? D 0:00 | zpool status 44566 ? S 0:00 CRON 44567 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44568 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44570 ? D 0:00 | zpool status 44579 ? S 0:00 CRON 44580 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 44581 ? S 0:00 | run-parts --report /etc/cron.hourly 44582 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 44584 ? D 0:00 | zpool status 44612 ? S 0:00 CRON 44613 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44614 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44616 ? D 0:00 | zpool status 44666 ? S 0:00 CRON 44667 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44668 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44670 ? D 0:00 | zpool status 44707 ? S 0:00 CRON 44708 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44709 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44711 ? D 0:00 | zpool status 44771 ? S 0:00 CRON 44772 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44773 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44775 ? D 0:00 | zpool status 44785 ? S 0:00 CRON 44786 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 44787 ? S 0:00 | run-parts --report /etc/cron.hourly 44788 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 44790 ? D 0:00 | zpool status 44829 ? S 0:00 CRON 44830 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44831 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44833 ? D 0:00 | zpool status 44898 ? S 0:00 CRON 44899 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44900 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44902 ? D 0:00 | zpool status 44934 ? S 0:00 CRON 44935 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44936 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 44938 ? D 0:00 | zpool status 45008 ? S 0:00 CRON 45009 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45010 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45012 ? D 0:00 | zpool status 45021 ? S 0:00 CRON 45022 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 45023 ? S 0:00 | run-parts --report /etc/cron.hourly 45024 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 45026 ? D 0:00 | zpool status 45058 ? S 0:00 CRON 45059 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45060 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45062 ? D 0:00 | zpool status 45166 ? S 0:00 CRON 45167 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45168 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45170 ? D 0:00 | zpool status 45202 ? S 0:00 CRON 45203 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45204 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45206 ? D 0:00 | zpool status 45252 ? S 0:00 CRON 45253 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45254 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45256 ? D 0:00 | zpool status 45262 ? S 0:00 CRON 45263 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 45264 ? S 0:00 | run-parts --report /etc/cron.hourly 45265 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 45267 ? D 0:00 | zpool status 45285 ? S 0:00 CRON 45286 ? Ss 0:00 | /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 45287 ? S 0:00 | run-parts --report /etc/cron.daily 45339 ? S 0:00 | /bin/bash /etc/cron.daily/clone 45340 ? D 0:00 | zpool status -x 45370 ? S 0:00 CRON 45371 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45372 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45374 ? D 0:00 | zpool status 45423 ? S 0:00 CRON 45424 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45425 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45427 ? D 0:00 | zpool status 45463 ? S 0:00 CRON 45464 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45465 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45467 ? D 0:00 | zpool status 45553 ? S 0:00 CRON 45554 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45555 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45557 ? D 0:00 | zpool status 45563 ? S 0:00 CRON 45564 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 45565 ? S 0:00 | run-parts --report /etc/cron.hourly 45566 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 45568 ? D 0:00 | zpool status 45596 ? S 0:00 CRON 45597 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45598 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45600 ? D 0:00 | zpool status 45662 ? S 0:00 CRON 45663 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45664 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45666 ? D 0:00 | zpool status 45710 ? S 0:00 CRON 45711 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45712 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45714 ? D 0:00 | zpool status 45788 ? S 0:00 CRON 45789 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45790 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45792 ? D 0:00 | zpool status 45798 ? S 0:00 CRON 45799 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 45800 ? S 0:00 | run-parts --report /etc/cron.hourly 45801 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 45803 ? D 0:00 | zpool status 45836 ? S 0:00 CRON 45837 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45838 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45840 ? D 0:00 | zpool status 45914 ? S 0:00 CRON 45915 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45916 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 45918 ? D 0:00 | zpool status 46024 ? S 0:00 CRON 46025 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46026 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46028 ? D 0:00 | zpool status 46093 ? S 0:00 CRON 46094 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46095 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46097 ? D 0:00 | zpool status 46107 ? S 0:00 CRON 46108 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 46109 ? S 0:00 | run-parts --report /etc/cron.hourly 46110 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 46112 ? D 0:00 | zpool status 46166 ? S 0:00 CRON 46167 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46168 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46170 ? D 0:00 | zpool status 46253 ? S 0:00 CRON 46254 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46255 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46257 ? D 0:00 | zpool status 46369 ? S 0:00 CRON 46370 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46371 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46373 ? D 0:00 | zpool status 46506 ? S 0:00 CRON 46507 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46508 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46510 ? D 0:00 | zpool status 46521 ? S 0:00 CRON 46522 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 46523 ? S 0:00 | run-parts --report /etc/cron.hourly 46524 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 46526 ? D 0:00 | zpool status 46618 ? S 0:00 CRON 46619 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46620 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46622 ? D 0:00 | zpool status 46763 ? S 0:00 CRON 46764 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46765 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46767 ? D 0:00 | zpool status 46887 ? S 0:00 CRON 46888 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46889 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 46891 ? D 0:00 | zpool status 47000 ? S 0:00 CRON 47001 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47002 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47004 ? D 0:00 | zpool status 47020 ? S 0:00 CRON 47021 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 47022 ? S 0:00 | run-parts --report /etc/cron.hourly 47023 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 47025 ? D 0:00 | zpool status 47135 ? S 0:00 CRON 47136 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47137 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47139 ? D 0:00 | zpool status 47289 ? S 0:00 CRON 47290 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47291 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47293 ? D 0:00 | zpool status 47378 ? S 0:00 CRON 47379 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47380 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47382 ? D 0:00 | zpool status 47517 ? S 0:00 CRON 47518 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47519 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47521 ? D 0:00 | zpool status 47542 ? S 0:00 CRON 47543 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 47544 ? S 0:00 | run-parts --report /etc/cron.hourly 47545 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 47547 ? D 0:00 | zpool status 47637 ? S 0:00 CRON 47638 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47639 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47641 ? D 0:00 | zpool status 47773 ? S 0:00 CRON 47774 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47775 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47777 ? D 0:00 | zpool status 47888 ? S 0:00 CRON 47889 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47890 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 47892 ? D 0:00 | zpool status 48006 ? S 0:00 CRON 48007 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48008 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48010 ? D 0:00 | zpool status 48044 ? S 0:00 CRON 48045 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 48046 ? S 0:00 | run-parts --report /etc/cron.hourly 48047 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 48049 ? D 0:00 | zpool status 48138 ? S 0:00 CRON 48139 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48140 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48142 ? D 0:00 | zpool status 48249 ? S 0:00 CRON 48250 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48251 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48253 ? D 0:00 | zpool status 48368 ? S 0:00 CRON 48369 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48370 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48372 ? D 0:00 | zpool status 48518 ? S 0:00 CRON 48519 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48520 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48522 ? D 0:00 | zpool status 48533 ? S 0:00 CRON 48534 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 48535 ? S 0:00 | run-parts --report /etc/cron.hourly 48536 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 48538 ? D 0:00 | zpool status 48642 ? S 0:00 CRON 48643 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48644 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48646 ? D 0:00 | zpool status 48773 ? S 0:00 CRON 48774 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48775 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48777 ? D 0:00 | zpool status 48921 ? S 0:00 CRON 48922 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48923 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 48925 ? D 0:00 | zpool status 49081 ? S 0:00 CRON 49082 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49083 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49085 ? D 0:00 | zpool status 49092 ? S 0:00 CRON 49093 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 49094 ? S 0:00 | run-parts --report /etc/cron.hourly 49095 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 49097 ? D 0:00 | zpool status 49193 ? S 0:00 CRON 49194 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49195 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49197 ? D 0:00 | zpool status 49333 ? S 0:00 CRON 49334 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49335 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49337 ? D 0:00 | zpool status 49455 ? S 0:00 CRON 49456 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49457 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49459 ? D 0:00 | zpool status 49593 ? S 0:00 CRON 49594 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49595 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49597 ? D 0:00 | zpool status 49609 ? S 0:00 CRON 49610 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 49611 ? S 0:00 | run-parts --report /etc/cron.hourly 49612 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 49614 ? D 0:00 | zpool status 49694 ? S 0:00 CRON 49695 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49696 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49698 ? D 0:00 | zpool status 49845 ? S 0:00 CRON 49846 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49847 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 49849 ? D 0:00 | zpool status 50008 ? S 0:00 CRON 50009 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50010 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50012 ? D 0:00 | zpool status 50160 ? S 0:00 CRON 50161 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50162 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50164 ? D 0:00 | zpool status 50180 ? S 0:00 CRON 50181 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 50182 ? S 0:00 | run-parts --report /etc/cron.hourly 50183 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 50185 ? D 0:00 | zpool status 50299 ? S 0:00 CRON 50300 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50301 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50303 ? D 0:00 | zpool status 50475 ? S 0:00 CRON 50476 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50477 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50479 ? D 0:00 | zpool status 50563 ? S 0:00 CRON 50565 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50566 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50568 ? D 0:00 | zpool status 50654 ? S 0:00 CRON 50655 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50656 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50658 ? D 0:00 | zpool status 50668 ? S 0:00 CRON 50669 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 50670 ? S 0:00 | run-parts --report /etc/cron.hourly 50671 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 50673 ? D 0:00 | zpool status 50750 ? S 0:00 CRON 50751 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50752 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50754 ? D 0:00 | zpool status 50858 ? S 0:00 CRON 50859 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50860 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50862 ? D 0:00 | zpool status 50928 ? S 0:00 CRON 50929 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50930 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50932 ? D 0:00 | zpool status 50976 ? S 0:00 CRON 50977 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50978 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 50980 ? D 0:00 | zpool status 50989 ? S 0:00 CRON 50990 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 50991 ? S 0:00 | run-parts --report /etc/cron.hourly 50992 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 50994 ? D 0:00 | zpool status 51032 ? S 0:00 CRON 51033 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51034 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51036 ? D 0:00 | zpool status 51176 ? S 0:00 CRON 51177 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51178 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51180 ? D 0:00 | zpool status 51343 ? S 0:00 CRON 51344 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51345 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51347 ? D 0:00 | zpool status 51414 ? S 0:00 CRON 51415 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51416 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51418 ? D 0:00 | zpool status 51424 ? S 0:00 CRON 51425 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 51426 ? S 0:00 | run-parts --report /etc/cron.hourly 51427 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 51429 ? D 0:00 | zpool status 51517 ? S 0:00 CRON 51518 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51519 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51521 ? D 0:00 | zpool status 51589 ? S 0:00 CRON 51590 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51591 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51593 ? D 0:00 | zpool status 51644 ? S 0:00 CRON 51645 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51646 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51648 ? D 0:00 | zpool status 51702 ? S 0:00 CRON 51703 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51704 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51706 ? D 0:00 | zpool status 51712 ? S 0:00 CRON 51713 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 51714 ? S 0:00 | run-parts --report /etc/cron.hourly 51715 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 51717 ? D 0:00 | zpool status 51750 ? S 0:00 CRON 51751 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51752 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51754 ? D 0:00 | zpool status 51851 ? S 0:00 CRON 51852 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51853 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51855 ? D 0:00 | zpool status 51894 ? S 0:00 CRON 51895 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51896 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51898 ? D 0:00 | zpool status 51954 ? S 0:00 CRON 51955 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51956 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 51958 ? D 0:00 | zpool status 51965 ? S 0:00 CRON 51966 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 51967 ? S 0:00 | run-parts --report /etc/cron.hourly 51968 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 51970 ? D 0:00 | zpool status 51999 ? S 0:00 CRON 52000 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52001 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52003 ? D 0:00 | zpool status 52091 ? S 0:00 CRON 52092 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52093 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52095 ? D 0:00 | zpool status 52143 ? S 0:00 CRON 52144 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52145 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52147 ? D 0:00 | zpool status 52191 ? S 0:00 CRON 52192 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52193 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52195 ? D 0:00 | zpool status 52201 ? S 0:00 CRON 52202 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 52203 ? S 0:00 | run-parts --report /etc/cron.hourly 52204 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 52206 ? D 0:00 | zpool status 52244 ? S 0:00 CRON 52245 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52246 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52248 ? D 0:00 | zpool status 52316 ? S 0:00 CRON 52317 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52318 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52320 ? D 0:00 | zpool status 52356 ? S 0:00 CRON 52357 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52358 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52360 ? D 0:00 | zpool status 52405 ? S 0:00 CRON 52406 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52407 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52409 ? D 0:00 | zpool status 52418 ? S 0:00 CRON 52419 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 52420 ? S 0:00 | run-parts --report /etc/cron.hourly 52421 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 52423 ? D 0:00 | zpool status 52467 ? S 0:00 CRON 52468 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52469 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52471 ? D 0:00 | zpool status 52595 ? S 0:00 CRON 52596 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52597 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52599 ? D 0:00 | zpool status 52647 ? S 0:00 CRON 52648 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52649 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52651 ? D 0:00 | zpool status 52714 ? S 0:00 CRON 52715 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52716 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52718 ? D 0:00 | zpool status 52728 ? S 0:00 CRON 52729 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 52730 ? S 0:00 | run-parts --report /etc/cron.hourly 52731 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 52733 ? D 0:00 | zpool status 52768 ? S 0:00 CRON 52769 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52770 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52772 ? D 0:00 | zpool status 52840 ? S 0:00 CRON 52841 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52842 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52844 ? D 0:00 | zpool status 52925 ? S 0:00 CRON 52926 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52927 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52929 ? D 0:00 | zpool status 52982 ? S 0:00 CRON 52983 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52984 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 52986 ? D 0:00 | zpool status 52998 ? S 0:00 CRON 52999 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 53000 ? S 0:00 | run-parts --report /etc/cron.hourly 53001 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 53003 ? D 0:00 | zpool status 53044 ? S 0:00 CRON 53045 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53046 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53048 ? D 0:00 | zpool status 53124 ? S 0:00 CRON 53125 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53126 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53128 ? D 0:00 | zpool status 53161 ? S 0:00 CRON 53162 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53163 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53165 ? D 0:00 | zpool status 53238 ? S 0:00 CRON 53239 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53240 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53242 ? D 0:00 | zpool status 53248 ? S 0:00 CRON 53249 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 53250 ? S 0:00 | run-parts --report /etc/cron.hourly 53251 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 53253 ? D 0:00 | zpool status 53284 ? S 0:00 CRON 53285 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53286 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53288 ? D 0:00 | zpool status 53356 ? S 0:00 CRON 53357 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53358 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53360 ? D 0:00 | zpool status 53404 ? S 0:00 CRON 53405 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53406 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53408 ? D 0:00 | zpool status 53456 ? S 0:00 CRON 53457 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53458 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53460 ? D 0:00 | zpool status 53466 ? S 0:00 CRON 53467 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 53468 ? S 0:00 | run-parts --report /etc/cron.hourly 53469 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 53471 ? D 0:00 | zpool status 53499 ? S 0:00 CRON 53500 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53501 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53503 ? D 0:00 | zpool status 53571 ? S 0:00 CRON 53572 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53573 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53575 ? D 0:00 | zpool status 53618 ? S 0:00 CRON 53619 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53620 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53622 ? D 0:00 | zpool status 53700 ? S 0:00 CRON 53701 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53702 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53704 ? D 0:00 | zpool status 53710 ? S 0:00 CRON 53711 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 53712 ? S 0:00 | run-parts --report /etc/cron.hourly 53713 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 53715 ? D 0:00 | zpool status 53752 ? S 0:00 CRON 53753 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53754 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53756 ? D 0:00 | zpool status 53816 ? S 0:00 CRON 53817 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53818 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53820 ? D 0:00 | zpool status 53859 ? S 0:00 CRON 53860 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53861 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53863 ? D 0:00 | zpool status 53911 ? S 0:00 CRON 53912 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53913 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 53915 ? D 0:00 | zpool status 53924 ? S 0:00 CRON 53925 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 53926 ? S 0:00 | run-parts --report /etc/cron.hourly 53927 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 53929 ? D 0:00 | zpool status 53947 ? S 0:00 CRON 53948 ? Ss 0:00 | /bin/sh -c test -x /usr/sbin/anacron || ( cd / && run-parts --report /etc/cron.daily ) 53949 ? S 0:00 | run-parts --report /etc/cron.daily 54001 ? S 0:00 | /bin/bash /etc/cron.daily/clone 54002 ? D 0:00 | zpool status -x 54014 ? S 0:00 CRON 54015 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54016 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54018 ? D 0:00 | zpool status 54094 ? S 0:00 CRON 54095 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54096 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54098 ? D 0:00 | zpool status 54133 ? S 0:00 CRON 54134 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54135 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54137 ? D 0:00 | zpool status 54187 ? S 0:00 CRON 54188 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54189 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54191 ? D 0:00 | zpool status 54200 ? S 0:00 CRON 54201 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 54202 ? S 0:00 | run-parts --report /etc/cron.hourly 54203 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 54205 ? D 0:00 | zpool status 54238 ? S 0:00 CRON 54239 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54240 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54242 ? D 0:00 | zpool status 54302 ? S 0:00 CRON 54303 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54304 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54306 ? D 0:00 | zpool status 54355 ? S 0:00 CRON 54356 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54357 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54359 ? D 0:00 | zpool status 54404 ? S 0:00 CRON 54405 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54406 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54408 ? D 0:00 | zpool status 54418 ? S 0:00 CRON 54419 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 54420 ? S 0:00 | run-parts --report /etc/cron.hourly 54421 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 54423 ? D 0:00 | zpool status 54454 ? S 0:00 CRON 54455 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54456 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54458 ? D 0:00 | zpool status 54522 ? S 0:00 CRON 54523 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54524 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54526 ? D 0:00 | zpool status 54570 ? S 0:00 CRON 54571 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54572 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54574 ? D 0:00 | zpool status 54696 ? S 0:00 CRON 54697 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54698 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54700 ? D 0:00 | zpool status 54710 ? S 0:00 CRON 54711 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 54712 ? S 0:00 | run-parts --report /etc/cron.hourly 54713 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 54715 ? D 0:00 | zpool status 54795 ? S 0:00 CRON 54796 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54797 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54799 ? D 0:00 | zpool status 54874 ? S 0:00 CRON 54875 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54876 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54878 ? D 0:00 | zpool status 54965 ? S 0:00 CRON 54966 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54967 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 54969 ? D 0:00 | zpool status 55054 ? S 0:00 CRON 55055 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55056 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55058 ? D 0:00 | zpool status 55083 ? S 0:00 CRON 55084 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 55085 ? S 0:00 | run-parts --report /etc/cron.hourly 55086 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 55088 ? D 0:00 | zpool status 55202 ? S 0:00 CRON 55203 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55204 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55206 ? D 0:00 | zpool status 55347 ? S 0:00 CRON 55348 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55349 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55351 ? D 0:00 | zpool status 55503 ? S 0:00 CRON 55504 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55505 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55507 ? D 0:00 | zpool status 55599 ? S 0:00 CRON 55600 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55601 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot -q -g --label=frequent --keep=4 // 55603 ? D 0:00 | zpool status 55613 ? S 0:00 CRON 55614 ? Ss 0:00 | /bin/sh -c cd / && run-parts --report /etc/cron.hourly 55615 ? S 0:00 | run-parts --report /etc/cron.hourly 55616 ? S 0:00 | /bin/sh /sbin/zfs-auto-snapshot --quiet --syslog --label=hourly --keep=8 // 55618 ? D 0:00 | zpool status 55720 ? S 0:00 CRON 55721 ? Ss 0:00 | /bin/sh -c zfs-auto-snapshot -q -g --label=freque

[mailinglists35]

a hint on what to do to help debug this would be appreciated.

[behlendorf]

It appears that you might have a couple issues going on here. But if your able to reproduce the issue and can still log in to the node (or have console) access it would be very helpful to dump all of the stacks. You can do this by issuing a sysrq-t either on the console or by echo t >/proc/sysrq_trigger. This will cause all the stack to be dumped to dmesg. That should show us what everything is blocked on.

[mailinglists35]

On Tue, Aug 7, 2012 at 8:58 PM, Brian Behlendorf notifications@github.comwrote:

It appears that you might have a couple issues going on here. But if your able to reproduce the issue and can still log in to the node (or have console) access it would be very helpful to dump all of the stacks. You can do this by issuing a sysrq-t either on the console or by echo t

/proc/sysrq_trigger. This will cause all the stack to be dumped to dmesg. That should show us what everything is blocked on.

— Reply to this email directly or view it on GitHubhttps://github.com/zfsonlinux/spl/issues/143#issuecomment-7561405.

Thank you very much for the hint. In the meantime the ubuntu ppa deb has been updated to 0.6.0.69-0ubuntu3~precise1 and I have disabled the "frequent" and "hourly" zfs-auto-snapshot cron runs, but I'll eventually re-enable it back so maybe it happens again. Btw, are there some stress tools available against zpool/zfs/spl?

[behlendorf]

Alright, certainly let me know if you hit the issue again.

As for stress tools I typically run the xfstests test suite which is what the upstream kernel folks use for ext4, xfs, etc. There are many more but we don't really have a zfs specific test suite for Linux just yet.

[mailinglists35]

as i'm tracking the daily builds on the ubuntu ppa, i'm now running 0.6.0.71-0ubuntu1~precise1

this got caught by the logcheck cron run this morning. Not sure if it is related. sysrq-t dumped 7 hours later at http://paste.ubuntu.com/1158784/. zpool/zfs seems to work ok, no processes in D state.

Aug 21 06:58:45 mailhost kernel: [1158944.802862] INFO: task zfs:51806 blocked for more than 120 seconds. Aug 21 06:58:45 mailhost kernel: [1158944.802890] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Aug 21 06:58:45 mailhost kernel: [1158944.802928] zfs D ffffffff81806200 0 51806 51769 0x00000000 Aug 21 06:58:45 mailhost kernel: [1158944.802934] ffff88003f187c98 0000000000000086 0000000000000000 0000000000000000 Aug 21 06:58:45 mailhost kernel: [1158944.802940] ffff88003f187fd8 ffff88003f187fd8 ffff88003f187fd8 0000000000013780 Aug 21 06:58:45 mailhost kernel: [1158944.802944] ffffffff81c0d020 ffff8800007cc500 ffff88003f187ca8 ffff88003b220258 Aug 21 06:58:45 mailhost kernel: [1158944.802949] Call Trace: Aug 21 06:58:45 mailhost kernel: [1158944.802959] [] schedule+0x3f/0x60 Aug 21 06:58:45 mailhost kernel: [1158944.802974] [] cv_wait_common+0x98/0x190 [spl] Aug 21 06:58:45 mailhost kernel: [1158944.802980] [] ? __wake_up+0x53/0x70 Aug 21 06:58:45 mailhost kernel: [1158944.802985] [] ? add_wait_queue+0x60/0x60 Aug 21 06:58:45 mailhost kernel: [1158944.802994] [] __cv_wait+0x13/0x20 [spl] Aug 21 06:58:45 mailhost kernel: [1158944.803032] [] txg_wait_synced+0xb3/0x190 [zfs] Aug 21 06:58:45 mailhost kernel: [1158944.803061] [] dsl_sync_task_group_wait+0x17d/0x2a0 [zfs] Aug 21 06:58:45 mailhost kernel: [1158944.803087] [] dmu_snapshots_destroy_nvl+0x14b/0x1e0 [zfs] Aug 21 06:58:45 mailhost kernel: [1158944.803119] [] zfs_ioc_destroy_snaps_nvl+0xf2/0x120 [zfs] Aug 21 06:58:45 mailhost kernel: [1158944.803148] [] ? spa_open+0x13/0x20 [zfs] Aug 21 06:58:45 mailhost kernel: [1158944.803179] [] zfsdev_ioctl+0xdc/0x1b0 [zfs] Aug 21 06:58:45 mailhost kernel: [1158944.803184] [] do_vfs_ioctl+0x8a/0x340 Aug 21 06:58:45 mailhost kernel: [1158944.803188] [] ? putname+0x35/0x50 Aug 21 06:58:45 mailhost kernel: [1158944.803191] [] ? do_sys_open+0x171/0x220 Aug 21 06:58:45 mailhost kernel: [1158944.803195] [] sys_ioctl+0x91/0xa0 Aug 21 06:58:45 mailhost kernel: [1158944.803200] [] ? do_device_not_available+0xe/0x10 Aug 21 06:58:45 mailhost kernel: [1158944.803204] [] system_call_fastpath+0x16/0x1b

[...other daemons...]

Aug 21 06:59:06 mailhost zfs-auto-snap: @zfs-auto-snap_daily-2012-08-21-0656, 1 created, 1 destroyed, 0 warnings.

[behlendorf]

My guess is this was just related to a lack of free memory on the machine delaying the txg_sync thread. These issues are being worked and I'm hopeful thing will be better for -rc11, but we shall see.

[mailinglists35]

I'm still getting occasional errors like these (actually only two of them, caught by logcheck): This happens when zfs-auto-snapshot cron script runs. Updated with latest ubuntu-zfs package.

Oct 9 07:03:07 mailhost kernel: [451101.047619] INFO: task zfs:5546 blocked for more than 120 seconds. Oct 9 07:03:07 mailhost kernel: [451101.047647] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 9 07:03:07 mailhost kernel: [451101.047685] zfs D ffffffff81806200 0 5546 5522 0x00000000 Oct 9 07:03:07 mailhost kernel: [451101.047690] ffff88003bce1c98 0000000000000082 0000000000000000 0000000000000000 Oct 9 07:03:07 mailhost kernel: [451101.047696] ffff88003bce1fd8 ffff88003bce1fd8 ffff88003bce1fd8 00000000000137c0 Oct 9 07:03:07 mailhost kernel: [451101.047701] ffffffff81c0d020 ffff88007583dc00 ffff88003bce1ca8 ffff880078f1a258 Oct 9 07:03:07 mailhost kernel: [451101.047706] Call Trace: Oct 9 07:03:07 mailhost kernel: [451101.047715] [] schedule+0x3f/0x60 Oct 9 07:03:07 mailhost kernel: [451101.047732] [] cv_wait_common+0x98/0x190 [spl] Oct 9 07:03:07 mailhost kernel: [451101.047738] [] ? __wake_up+0x53/0x70 Oct 9 07:03:07 mailhost kernel: [451101.047743] [] ? add_wait_queue+0x60/0x60 Oct 9 07:03:07 mailhost kernel: [451101.047752] [] __cv_wait+0x13/0x20 [spl] Oct 9 07:03:07 mailhost kernel: [451101.047792] [] txg_wait_synced+0xb3/0x190 [zfs] Oct 9 07:03:07 mailhost kernel: [451101.047822] [] dsl_sync_task_group_wait+0x17d/0x2a0 [zfs] Oct 9 07:03:07 mailhost kernel: [451101.047847] [] dmu_snapshots_destroy_nvl+0x14b/0x1e0 [zfs] Oct 9 07:03:07 mailhost kernel: [451101.047878] [] zfs_ioc_destroy_snaps_nvl+0xfa/0x130 [zfs] Oct 9 07:03:07 mailhost kernel: [451101.047908] [] ? spa_open+0x13/0x20 [zfs] Oct 9 07:03:07 mailhost kernel: [451101.047938] [] zfsdev_ioctl+0xdc/0x1b0 [zfs] Oct 9 07:03:07 mailhost kernel: [451101.047943] [] do_vfs_ioctl+0x8a/0x340 Oct 9 07:03:07 mailhost kernel: [451101.047947] [] ? putname+0x35/0x50 Oct 9 07:03:07 mailhost kernel: [451101.047950] [] ? do_sys_open+0x171/0x220 Oct 9 07:03:07 mailhost kernel: [451101.047954] [] sys_ioctl+0x91/0xa0 Oct 9 07:03:07 mailhost kernel: [451101.047959] [] system_call_fastpath+0x16/0x1b

Oct 11 06:55:07 mailhost kernel: [623383.021135] INFO: task zfs:32526 blocked for more than 120 seconds. Oct 11 06:55:07 mailhost kernel: [623383.021163] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Oct 11 06:55:07 mailhost kernel: [623383.021201] zfs D ffffffff81806200 0 32526 32503 0x00000000 Oct 11 06:55:07 mailhost kernel: [623383.021207] ffff880006251c98 0000000000000082 0000000000000000 0000000000000000 Oct 11 06:55:07 mailhost kernel: [623383.021213] ffff880006251fd8 ffff880006251fd8 ffff880006251fd8 00000000000137c0 Oct 11 06:55:07 mailhost kernel: [623383.021218] ffff880078c21700 ffff88007754c500 ffff880006251ca8 ffff880078f1a258 Oct 11 06:55:07 mailhost kernel: [623383.021223] Call Trace: Oct 11 06:55:07 mailhost kernel: [623383.021233] [] schedule+0x3f/0x60 Oct 11 06:55:07 mailhost kernel: [623383.021250] [] cv_wait_common+0x98/0x190 [spl] Oct 11 06:55:07 mailhost kernel: [623383.021256] [] ? __wake_up+0x53/0x70 Oct 11 06:55:07 mailhost kernel: [623383.021261] [] ? add_wait_queue+0x60/0x60 Oct 11 06:55:07 mailhost kernel: [623383.021270] [] __cv_wait+0x13/0x20 [spl] Oct 11 06:55:07 mailhost kernel: [623383.021310] [] txg_wait_synced+0xb3/0x190 [zfs] Oct 11 06:55:07 mailhost kernel: [623383.021341] [] dsl_sync_task_group_wait+0x17d/0x2a0 [zfs] Oct 11 06:55:07 mailhost kernel: [623383.021367] [] dmu_snapshots_destroy_nvl+0x14b/0x1e0 [zfs] Oct 11 06:55:07 mailhost kernel: [623383.021399] [] zfs_ioc_destroy_snaps_nvl+0xfa/0x130 [zfs] Oct 11 06:55:07 mailhost kernel: [623383.021430] [] ? spa_open+0x13/0x20 [zfs] Oct 11 06:55:07 mailhost kernel: [623383.021460] [] zfsdev_ioctl+0xdc/0x1b0 [zfs] Oct 11 06:55:07 mailhost kernel: [623383.021465] [] do_vfs_ioctl+0x8a/0x340 Oct 11 06:55:07 mailhost kernel: [623383.021468] [] ? putname+0x35/0x50 Oct 11 06:55:07 mailhost kernel: [623383.021472] [] ? do_sys_open+0x171/0x220 Oct 11 06:55:07 mailhost kernel: [623383.021476] [] sys_ioctl+0x91/0xa0 Oct 11 06:55:07 mailhost kernel: [623383.021481] [] system_call_fastpath+0x16/0x1b

[mailinglists35]

At 6:25 am everyday, run-parts called from crontab kicks cron.daily scripts. there is a rsync script that mirrors the whole os directory tree to the zfs directory, then zfs-auto-snapshot runs. however, looking at munin graphs apparently there is enough memory: http://postimage.org/image/djczf0kt3/

[behlendorf]

@mailinglists35 Does anything fail? The stacks you posted are just advisory and I'm seriously considering suppressing them to avoid them causing unwarranted concern.

[mailinglists35]

No, I have noticed nothing else failing. I'm certainly sleeping anyway at the time cron.daily runs :) and so the users of that box. If that kernel message means just that zfs needs a little more time to complete it's action then I shall not worry about it. But if you suppress the message then maybe the faq entry "1.10 What’s going on with performance?" wouldn't need this added? something like "under certain conditions (less than 2gb ram?), you might experience zfs process in D state for a couple of minutes"

[mailinglists35]

i have narrowed this down: evey time zfs command is run when another zfs is creating a snapshot, results in permanent hang. I hope this is reproductible for you too, as I can reproduce this on latest ubuntu 12.04 lts and debian wheezy kernels (3.2.0 amd64) with latest ubuntu-zfs packages. maybe /sbin/zfs is assuming it is the only one talking to the pool so when two meet, party begins? or on the other hand is spl unable to satisfy two zfs commands at once? i get no spl crash/dump. sysrq-t pasted at the end. hope this helps you determine where the issue belongs, zfs or spl

to reproduce: while true; do zfs list; zpool list; done open another terminal: zfs snapshot poolname@test1 result: both zfs commands hang

Dec 3 19:29:03 homerouter kernel: [ 427.749244] SPL: Loaded module v0.6.0.88-rc12 Dec 3 19:29:03 homerouter kernel: [ 427.784978] ZFS: Loaded module v0.6.0.88-rc12, ZFS pool version 28, ZFS filesystem version 5

root@homerouter:~# zfs list NAME USED AVAIL REFER MOUNTPOINT homebackup 743G 171G 144K /mnt/seagate910 homebackup/homerouter 15.4G 171G 15.1G /mnt/seagate910/homerouter homebackup/ntfs 727G 171G 713G - root@homerouter:~# ls /mnt/seagate910/ homerouter root@homerouter:~# ls /mnt/seagate910/homerouter/ bin dev initrd.img lib64 mnt root selinux sysrcd.dat var boot etc lib lost+found opt run srv tmp vmlinuz cgroup home lib32 media proc sbin sys usr root@homerouter:~# ls /mnt/seagate910/homerouter/.zfs/ shares snapshot root@homerouter:~# ls /mnt/seagate910/homerouter/.zfs/snapshot/ [...](similar snapshot names cut) [...] zfs-auto-snap_weekly-2012-11-25-0647 zfs-auto-snap_weekly-2012-12-02-0647 root@homerouter:~# ls /mnt/seagate910/homerouter/.zfs/snapshot/zfs-auto-snap_weekly-2012-12-02-0647 bin dev initrd.img lib64 mnt root selinux sysrcd.dat var boot etc lib lost+found opt run srv tmp vmlinuz cgroup home lib32 media proc sbin sys usr root@homerouter:~# ls /mnt/seagate910/homerouter/.zfs/snapshot/zfs-auto-snap_weekly-2012-11-04-0747 bin dev initrd.img lib64 mnt root selinux sysrcd.dat var boot etc lib lost+found opt run srv tmp vmlinuz cgroup home lib32 media proc sbin sys usr root@homerouter:~# ls /mnt/seagate910/homerouter/.zfs/snapshot/zfs-auto-snap_monthly-2012-11-01-0652 bin dev initrd.img lib64 mnt root selinux sysrcd.dat var boot etc lib lost+found opt run srv tmp vmlinuz cgroup home lib32 media proc sbin sys usr root@homerouter:~# ls /mnt/seagate910/homerouter/.zfs/snapshot/zfs-auto-snap_monthly-2012-11-01-0652/ bin/ home/ lost+found/ root/ sys/ vmlinuz boot/ initrd.img media/ run/ sysrcd.dat cgroup/ lib/ mnt/ sbin/ tmp/ dev/ lib32/ opt/ selinux/ usr/ etc/ lib64/ proc/ srv/ var/ root@homerouter:~# ls /mnt/seagate910/homerouter/.zfs/snapshot/zfs-auto-snap_monthly-2012-11-01-0652/usr/ bin include lib32 local sbin src usbview_x86_64.tar.gz games lib lib64 man share usbview_x86_64 root@homerouter:~# zfs list ( enter key, <- GAME OVER)

root@homerouter:~# jobs [1]+ Running tail -f /var/log/kern.log & root@homerouter:~# ps ax|grep D PID TTY STAT TIME COMMAND 3918 ? Ss 0:00 /usr/sbin/nmbd -D 3924 ? S 0:00 /usr/sbin/nmbd -D 4047 ? Ss 0:00 /usr/sbin/smbd -D 4115 ? S 0:00 /usr/sbin/smbd -D 15139 ? D< 0:00 [txg_sync] 16261 ? D 0:00 zfs snapshot -o com.sun:auto-snapshot-desc - -r homebackup@zfs-auto-snap_frequent-2012-12-03-1930 16723 pts/0 D+ 0:00 zfs list 16725 pts/1 S+ 0:00 grep D

(two minutes later, cpu idle but load begins to climb: %Cpu(s): 0.0 us, 0.2 sy, 0.0 ni, 99.8 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st top - 19:37:36 up 15 min, 2 users, load average: 3.84, 2.93, 1.65 system usable but zfs/zpool not)

root@homerouter:~# Dec 3 19:33:56 homerouter kernel: [ 720.612081] INFO: task txg_sync:15139 blocked for more than 120 seconds. Dec 3 19:33:56 homerouter kernel: [ 720.612128] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 19:33:56 homerouter kernel: [ 720.612190] txg_sync D ffff8800b8154400 0 15139 2 0x00000000 Dec 3 19:33:56 homerouter kernel: [ 720.612275] ffff8800b8154400 0000000000000046 0000000000000001 ffff8801156bf690 Dec 3 19:33:56 homerouter kernel: [ 720.612408] 0000000000013780 ffff8800b84d5fd8 ffff8800b84d5fd8 ffff8800b8154400 Dec 3 19:33:56 homerouter kernel: [ 720.612541] ffffffff810363d8 ffff8800b6cdc408 0000000000000000 ffff88010ccf6a40 Dec 3 19:33:56 homerouter kernel: [ 720.612674] Call Trace: Dec 3 19:33:56 homerouter kernel: [ 720.612717] [] ? should_resched+0x5/0x23 Dec 3 19:33:56 homerouter kernel: [ 720.612763] [] ? rwsem_down_failed_common+0xe0/0x114 Dec 3 19:33:56 homerouter kernel: [ 720.612811] [] ? call_rwsem_down_write_failed+0x13/0x20 Dec 3 19:33:56 homerouter kernel: [ 720.612858] [] ? down_write+0x25/0x27 Dec 3 19:33:56 homerouter kernel: [ 720.612926] [] ? zap_get_leaf_byblk+0x141/0x19c [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.612991] [] ? zap_deref_leaf+0x5e/0x67 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613056] [] ? fzap_add_cd+0x45/0x101 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613119] [] ? zap_hashbits+0x5/0x14 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613182] [] ? zap_hash+0x9b/0xb1 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613244] [] ? zap_name_alloc+0xba/0xce [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613308] [] ? zap_add+0xcb/0x122 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613368] [] ? dsl_dataset_snapshot_sync+0x3cd/0x466 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613448] [] ? dsl_props_set_sync+0x194/0x1af [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613507] [] ? snapshot_sync+0x1d/0xb2 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613565] [] ? snapshot_check+0x16/0xa8 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613629] [] ? dsl_sync_task_group_sync+0x11a/0x198 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613706] [] ? dsl_pool_sync+0x1df/0x402 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613772] [] ? spa_sync+0x47a/0x836 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613817] [] ? ktime_get_ts+0x5c/0x82 Dec 3 19:33:56 homerouter kernel: [ 720.613880] [] ? txg_sync_thread+0x2bd/0x49a [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613946] [] ? txg_thread_wait.isra.2+0x23/0x23 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.613996] [] ? thread_generic_wrapper+0x6a/0x75 [spl] Dec 3 19:33:56 homerouter kernel: [ 720.614046] [] ? thread_create+0x2be/0x2be [spl] Dec 3 19:33:56 homerouter kernel: [ 720.614092] [] ? kthread+0x76/0x7e Dec 3 19:33:56 homerouter kernel: [ 720.614136] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:33:56 homerouter kernel: [ 720.614182] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:33:56 homerouter kernel: [ 720.614227] [] ? gs_change+0x13/0x13 Dec 3 19:33:56 homerouter kernel: [ 720.614272] INFO: task zfs:16261 blocked for more than 120 seconds. Dec 3 19:33:56 homerouter kernel: [ 720.614316] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 19:33:56 homerouter kernel: [ 720.614376] zfs D ffff88011bc93780 0 16261 16013 0x00000000 Dec 3 19:33:56 homerouter kernel: [ 720.614460] ffff880113ab6200 0000000000000086 0000000000000000 ffff880116f080c0 Dec 3 19:33:56 homerouter kernel: [ 720.614593] 0000000000013780 ffff8800b4929fd8 ffff8800b4929fd8 ffff880113ab6200 Dec 3 19:33:56 homerouter kernel: [ 720.614727] 0000000000000202 000000018134cb14 ffff880108add290 ffff880108add288 Dec 3 19:33:56 homerouter kernel: [ 720.614861] Call Trace: Dec 3 19:33:56 homerouter kernel: [ 720.614902] [] ? cv_wait_common+0xcb/0x15e [spl] Dec 3 19:33:56 homerouter kernel: [ 720.614947] [] ? add_wait_queue+0x3c/0x3c Dec 3 19:33:56 homerouter kernel: [ 720.615011] [] ? txg_wait_synced+0x12b/0x168 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.615076] [] ? dsl_sync_task_group_wait+0x18e/0x1b8 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.616806] [] ? dmu_objset_snapshot+0x177/0x264 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.616871] [] ? zfs_ioc_snapshot+0x1a8/0x1d9 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.616936] [] ? zfsdev_ioctl+0x103/0x159 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.616984] [] ? do_vfs_ioctl+0x459/0x49a Dec 3 19:33:56 homerouter kernel: [ 720.617030] [] ? remove_vma+0x64/0x6b Dec 3 19:33:56 homerouter kernel: [ 720.617080] [] ? do_munmap+0x2da/0x2f3 Dec 3 19:33:56 homerouter kernel: [ 720.617130] [] ? sys_ioctl+0x4b/0x72 Dec 3 19:33:56 homerouter kernel: [ 720.617180] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:33:56 homerouter kernel: [ 720.617232] INFO: task zfs:16723 blocked for more than 120 seconds. Dec 3 19:33:56 homerouter kernel: [ 720.617280] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 19:33:56 homerouter kernel: [ 720.617346] zfs D ffff8800b313f810 0 16723 6328 0x00000000 Dec 3 19:33:56 homerouter kernel: [ 720.617350] ffff8800b313f810 0000000000000082 ffff880108bbe6c0 ffff88010aef2240 Dec 3 19:33:56 homerouter kernel: [ 720.617353] 0000000000013780 ffff880113965fd8 ffff880113965fd8 ffff8800b313f810 Dec 3 19:33:56 homerouter kernel: [ 720.617357] ffffffff810ebc21 ffff880100000394 ffffffffa071701d ffff880108add4e8 Dec 3 19:33:56 homerouter kernel: [ 720.617360] Call Trace: Dec 3 19:33:56 homerouter kernel: [ 720.617365] [] ? kmalloc+0x100/0x112 Dec 3 19:33:56 homerouter kernel: [ 720.617388] [] ? spa_name_compare+0x6/0x1b [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.617391] [] ? rwsem_down_failed_common+0xe0/0x114 Dec 3 19:33:56 homerouter kernel: [ 720.617395] [] ? call_rwsem_down_read_failed+0x14/0x30 Dec 3 19:33:56 homerouter kernel: [ 720.617398] [] ? down_read+0x17/0x19 Dec 3 19:33:56 homerouter kernel: [ 720.617417] [] ? dsl_dir_open_spa+0x174/0x42a [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.617440] [] ? spa_name_compare+0x6/0x1b [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.617444] [] ? avl_find+0x30/0x70 [zavl] Dec 3 19:33:56 homerouter kernel: [ 720.617462] [] ? dsl_dataset_hold+0x2d/0x20f [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.617465] [] ? mutex_lock+0xd/0x2d Dec 3 19:33:56 homerouter kernel: [ 720.617488] [] ? spa_open_common+0x2a6/0x2be [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.617504] [] ? dmu_objset_hold+0x16/0x47 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.617524] [] ? zfs_ioc_objset_stats+0x22/0x52 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.617544] [] ? zfsdev_ioctl+0x103/0x159 [zfs] Dec 3 19:33:56 homerouter kernel: [ 720.617547] [] ? do_vfs_ioctl+0x459/0x49a Dec 3 19:33:56 homerouter kernel: [ 720.617550] [] ? remove_vma+0x64/0x6b Dec 3 19:33:56 homerouter kernel: [ 720.617553] [] ? do_munmap+0x2da/0x2f3 Dec 3 19:33:56 homerouter kernel: [ 720.617556] [] ? sys_ioctl+0x4b/0x72 Dec 3 19:33:56 homerouter kernel: [ 720.617560] [] ? system_call_fastpath+0x16/0x1b

root@homerouter:~# echo t >/proc/sysrq-trigger root@homerouter:~# Dec 3 19:49:36 homerouter kernel: ] [] ? schedule_timeout+0x2c/0xdb Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? input_available_p+0xe/0x47 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? is_ignored+0x36/0x36 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? n_tty_read+0x3d5/0x72d Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? tty_poll+0x6f/0x6f Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? tty_read+0x88/0xc3 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? vfs_read+0x9f/0xe6 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_read+0x45/0x6b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] getty S ffff88011bc13780 0 6321 1 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880115e3c3c0 0000000000000082 0000000000000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff8801077f1fd8 ffff8801077f1fd8 ffff880115e3c3c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffffffff810363d8 000000018134cb3a ffff88011bc0e640 7fffffffffffffff Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_timeout+0x2c/0xdb Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? input_available_p+0xe/0x47 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? is_ignored+0x36/0x36 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? n_tty_read+0x3d5/0x72d Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? tty_poll+0x6f/0x6f Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? tty_read+0x88/0xc3 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? vfs_read+0x9f/0xe6 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_read+0x45/0x6b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] pppd S ffff88011bc13780 0 6322 1 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8801133848b0 0000000000000086 ffff880100000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff880108d7bfd8 ffff880108d7bfd8 ffff8801133848b0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000000286 0000000100000000 ffff880108d7bd58 0000000000000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_hrtimeout_range_clock+0xc0/0x103 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? update_rmtp+0x62/0x62 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? hrtimer_start_expires+0x13/0x15 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_schedule_timeout+0x43/0x5e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_select+0x429/0x46a Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_freewait+0x97/0x97 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? alloc_pages_nodemask+0x186/0x7aa Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? number.isra.1+0x10e/0x218 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sigqueue_alloc+0x11/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? fsnotify_clear_marks_by_inode+0x23/0xc2 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? swiotlb_tbl_map_single+0x1ce/0x1e0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? swiotlb_map_page+0x95/0xdb Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? e100_exec_cmd+0x9c/0xaa [e100] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pci_map_single+0x81/0x81 [e100] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? e100_exec_cb+0xff/0x110 [e100] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _cond_resched+0x7/0x1c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? core_sys_select+0x175/0x21a Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kmem_cache_free+0x2d/0x69 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? ppp_read+0x17d/0x18f [ppp_generic] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? timespec_add_safe+0x32/0x63 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_select+0x88/0xad Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] pppd S ffff88011bc93780 0 6323 1 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010c445650 0000000000000082 ffff880100000000 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cdd1fd8 ffff88010cdd1fd8 ffff88010c445650 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000000286 0000000100000000 ffff88010cdd1d58 0000000000000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_hrtimeout_range_clock+0xc0/0x103 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? update_rmtp+0x62/0x62 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? hrtimer_start_expires+0x13/0x15 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_schedule_timeout+0x43/0x5e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_select+0x429/0x46a Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_freewait+0x97/0x97 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? qh_append_tds+0x42b/0x441 [ehci_hcd] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? qh_link_async+0xb9/0xca [ehci_hcd] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? ehci_urb_enqueue+0x123/0xbc3 [ehci_hcd] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? swiotlb_map_page+0x95/0xdb Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? dma_map_single_attrs.constprop.24+0x65/0x71 [usbcore] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? dma_mapping_error+0x12/0x38 [usbcore] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? usb_hcd_map_urb_for_dma+0x1ff/0x275 [usbcore] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? usb_alloc_urb+0x11/0x36 [usbcore] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? usb_hcd_submit_urb+0x528/0x5e9 [usbcore] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? usbnet_start_xmit+0x26d/0x2a6 [usbnet] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _cond_resched+0x7/0x1c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? core_sys_select+0x175/0x21a Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kmem_cache_free+0x2d/0x69 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? ppp_read+0x17d/0x18f [ppp_generic] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? timespec_add_safe+0x32/0x63 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_select+0x88/0xad Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] bash S ffff88011bc93780 0 6328 6121 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880113b76970 0000000000000082 00007fffffffeffd ffff880116fdaf60 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff8801093dffd8 ffff8801093dffd8 ffff880113b76970 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 00000000015ab888 ffffffff810709c1 0000000000000246 ffff8801093dff00 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_wait+0x19b/0x20f Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_wait4+0x98/0xb5 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? fput+0x17a/0x1a1 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_stopped_code+0x37/0x37 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] sshd S ffff88011bc93780 0 6421 5961 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010cfaf1e0 0000000000000082 0000000000000000 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff880108e77fd8 ffff880108e77fd8 ffff88010cfaf1e0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010d9bf000 0000000100000001 ffff880108e77ae8 0000000000000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_hrtimeout_range_clock+0x47/0x103 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? wake_up+0x35/0x46 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_schedule_timeout+0x43/0x5e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_select+0x429/0x46a Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_freewait+0x97/0x97 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? test_tsk_need_resched+0xa/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? ttwu_do_wakeup+0x50/0xc4 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sock_aio_write+0x121/0x135 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _cond_resched+0x7/0x1c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? core_sys_select+0x175/0x21a Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_sync_write+0xb4/0xec Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? fsnotify+0x1e6/0x20c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_select+0x88/0xad Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_write+0x5f/0x6b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] bash R running task 0 6511 6421 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000002212408 ffff8800da5e9380 ffff8800da5e9380 0000000002212408 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffffffff810f9b41 ffff88010c494400 0000000000000000 ffff8800da5e9380 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000002212408 00007fdf18d757a0 ffffffff810f9d1e ffff88010c494400 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? vfs_write+0xa2/0xe9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_write+0x45/0x6b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] tail S ffff88011bc13780 0 6821 6511 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880113113060 0000000000000082 ffff8801132dd6c0 ffff880116f087b0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010778dfd8 ffff88010778dfd8 ffff880113113060 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000000246 ffffffff8134cb14 ffffffff810363d8 ffff8801134ff8d0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? inotify_read+0x237/0x28e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? add_wait_queue+0x3c/0x3c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? vfs_read+0x9f/0xe6 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_read+0x45/0x6b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ntpd S ffff88011bc13780 0 6868 1 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010bb17750 0000000000000086 ffff88011bc0e9c0 ffff880116f087b0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff880114915fd8 ffff880114915fd8 ffff88010bb17750 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000000286 0000000000000000 ffff880114915f08 0000000000000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_hrtimeout_range_clock+0xc0/0x103 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? update_rmtp+0x62/0x62 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? hrtimer_start_expires+0x13/0x15 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_schedule_timeout+0x43/0x5e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_sys_poll+0x2b9/0x347 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_freewait+0x97/0x97 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? inet_recvmsg+0x5b/0x6f Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? cpumask_intersects+0xb/0x12 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? select_task_rq_fair+0x422/0x679 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? check_preempt_wakeup+0xf8/0x17a Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? test_tsk_need_resched+0xa/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? check_preempt_curr+0x52/0x5f Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? ttwu_do_wakeup+0x50/0xc4 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x187/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? queue_work+0x224/0x24c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_adjtimex+0x501/0x50c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? _cond_resched+0x7/0x1c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_adjtimex+0x4a/0x62 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? timespec_add_safe+0x32/0x63 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_poll+0x43/0xab Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ntpd S ffff88011bc13780 0 6870 1 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8801096108f0 0000000000000086 ffff880100000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010931ffd8 ffff88010931ffd8 ffff8801096108f0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000000286 0000000100000000 ffff88010931ff08 0000000000000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_hrtimeout_range_clock+0xc0/0x103 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? update_rmtp+0x62/0x62 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? hrtimer_start_expires+0x13/0x15 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_schedule_timeout+0x43/0x5e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sock_poll_wait+0x15/0x15 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_sys_poll+0x2b9/0x347 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? poll_freewait+0x97/0x97 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? pollwait+0xce/0xce Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sock_aio_write+0x121/0x135 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? select_task_rq_fair+0x422/0x679 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sock_aio_dtor+0x9/0x9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_sync_readv_writev+0xa3/0xe0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? ttwu_do_wakeup+0x50/0xc4 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? fsnotify+0x1e6/0x20c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? do_readv_writev+0x107/0x11c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? timespec_add_safe+0x32/0x63 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_poll+0x43/0xab Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] pickup S ffff88011bc93780 0 6893 6051 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010bb16280 0000000000000082 ffff880100000000 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff8801092e9fd8 ffff8801092e9fd8 ffff88010bb16280 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88011bc8e9d0 0000000100000000 ffff8801092e9f38 0000000000000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_hrtimeout_range_clock+0xc0/0x103 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? update_rmtp+0x62/0x62 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? hrtimer_start_expires+0x13/0x15 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_epoll_wait+0x1b7/0x308 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] qmgr S ffff88010f812af0 0 6894 6051 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010f812af0 0000000000000086 ffff88011bc8e9c0 ffff88010bb16280 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010749ffd8 ffff88010749ffd8 ffff88010f812af0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010770cac0 0000000000000000 ffff88010749ff38 0000000000000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_hrtimeout_range_clock+0xc0/0x103 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? update_rmtp+0x62/0x62 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? hrtimer_start_expires+0x13/0x15 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? sys_epoll_wait+0x1b7/0x308 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? system_call_fastpath+0x16/0x1b Dec 3 19:49:36 homerouter kernel: [ 1660.691658] spl_system_task S ffff88011bc13780 0 14946 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880113034a30 0000000000000046 ffff880100000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff880114f1bfd8 ffff880114f1bfd8 ffff880113034a30 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010dfe30a8 00000001810709c1 0000000000000002 ffff88010dfe3080 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x88/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] spl_system_task S ffff88010c444870 0 14947 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010c444870 0000000000000046 ffff88010f223b40 ffff8801088b7690 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff880113409fd8 ffff880113409fd8 ffff88010c444870 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010dfe30a8 ffffffff810709c1 0000000000000002 ffff88010dfe3080 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x88/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] arc_adapt S ffff88011bc93780 0 14948 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b6db87b0 0000000000000046 0000000000000000 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc27fd8 ffff88010cc27fd8 ffff8800b6db87b0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000000282 0000000100000286 0000000000000010 00000001000530e8 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_timeout+0xa3/0xdb Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? usleep_range+0x3e/0x3e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? cv_timedwait_common+0x12a/0x1df [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? add_wait_queue+0x3c/0x3c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arc_adjust_meta+0x171/0x171 [zfs] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arc_adapt_thread+0x86/0xb4 [zfs] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? thread_generic_wrapper+0x6a/0x75 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? thread_create+0x2be/0x2be [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] l2arc_feed S ffff88011bc13780 0 14949 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88011364aea0 0000000000000046 0000000000000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc29fd8 ffff88010cc29fd8 ffff88011364aea0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000000286 0000000100000282 0000000000000002 00000001000530eb Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? schedule_timeout+0xa3/0xdb Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? usleep_range+0x3e/0x3e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? __cv_timedwait_common+0x12a/0x1df [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? should_resched+0x5/0x23 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? add_wait_queue+0x3c/0x3c Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? l2arc_feed_thread+0x50/0x698 [zfs] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? update_curr+0xbd/0xd7 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? l2arc_evict+0x227/0x227 [zfs] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? thread_generic_wrapper+0x6a/0x75 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? thread_create+0x2be/0x2be [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/0 S ffff8800b8749120 0 14950 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b8749120 0000000000000046 0000000000000000 ffff8800b76b2f20 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc2dfd8 ffff88010cc2dfd8 ffff8800b8749120 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x88/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/1 S ffff88011bc93780 0 14951 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010aef3710 0000000000000046 0000000000000000 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc2ffd8 ffff88010cc2ffd8 ffff88010aef3710 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 00000001810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/2 S ffff88011bc13780 0 14952 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8801133ad810 0000000000000046 0000000000000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc31fd8 ffff88010cc31fd8 ffff8801133ad810 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 00000001810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/3 S ffff8801133841c0 0 14953 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8801133841c0 0000000000000046 0000000000000000 ffff8800b64662c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc33fd8 ffff88010cc33fd8 ffff8801133841c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/4 S ffff88011bc13780 0 14954 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8801156befa0 0000000000000046 0000000000000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc35fd8 ffff88010cc35fd8 ffff8801156befa0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/5 S ffff88011bc93780 0 14955 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880116e9b550 0000000000000046 0000000000000001 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc37fd8 ffff88010cc37fd8 ffff880116e9b550 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/6 S ffff88011bc13780 0 14956 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880109610200 0000000000000046 0000000000000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc39fd8 ffff88010cc39fd8 ffff880109610200 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/7 S ffff88011bc93780 0 14957 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880036ebc240 0000000000000046 0000000000000001 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc3bfd8 ffff88010cc3bfd8 ffff880036ebc240 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/8 S ffff880113385690 0 14958 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880113385690 0000000000000046 0000000000000000 ffff8800b76b2f20 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc3dfd8 ffff88010cc3dfd8 ffff880113385690 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/9 S ffff88011bc93780 0 14959 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b7b93510 0000000000000046 0000000000000001 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010cc3ffd8 ffff88010cc3ffd8 ffff8800b7b93510 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/10 S ffff88011bc13780 0 14960 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010cf89610 0000000000000046 0000000000000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010fe81fd8 ffff88010fe81fd8 ffff88010cf89610 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 00000001810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/11 S ffff88011bc93780 0 14961 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880113034340 0000000000000046 0000000000000001 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010fe83fd8 ffff88010fe83fd8 ffff880113034340 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/12 S ffff88011bc13780 0 14962 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff880113035120 0000000000000046 0000000000000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010fe85fd8 ffff88010fe85fd8 ffff880113035120 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/13 S ffff88011bc93780 0 14963 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b81558d0 0000000000000046 0000000000000001 ffff880116f080c0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010fe87fd8 ffff88010fe87fd8 ffff8800b81558d0 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kernel_thread_helper+0x4/0x10 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread_worker_fn+0x139/0x139 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? gs_change+0x13/0x13 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] zvol/14 S ffff88011bc13780 0 14964 2 0x00000000 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff88010a074380 0000000000000046 0000000000000000 ffffffff8160d020 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] 0000000000013780 ffff88010fe89fd8 ffff88010fe89fd8 ffff88010a074380 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] ffff8800b4f345a8 ffffffff810709c1 0000000000000002 ffff8800b4f34580 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] Call Trace: Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? arch_local_irq_save+0x11/0x17 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? taskq_thread+0x148/0x4b2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? finish_task_switch+0x4e/0xb9 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? try_to_wake_up+0x197/0x197 Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? task_done+0xf2/0xf2 [spl] Dec 3 19:49:36 homerouter kernel: [ 1660.691658] [] ? kthread+0x76/0x7e Dec 3 19:49:36 homerouter kernel: [

[mailinglists35]

while (unsuccessfully) trying to reproduce #97 on this debian machine, I ran the diff -qr command on a snapdir. I'm not saying there is a connection between these two issues but that was just how I ran into it. This has happening when zfs-auto-snapshot kicked into creating a snapshot.

So it appears I cannot even read data from a snapshot dir while another snapshot gets created - Or - cannot create a snapshot while reading data from a previous snapshot; either way, cannot perform both actions at the same time.

(btw, zfs-auto-snapshot is also deleting older snapshots, but it did not reach the deletion step (it is blocked at creation) and anyway I was reading from a snapshot that was not scheduled to be deleted anyway)

root@homerouter:~# ps ax|grep D PID TTY STAT TIME COMMAND 4049 ? Ss 0:00 /usr/sbin/nmbd -D 4050 ? S 0:00 /usr/sbin/nmbd -D 4086 ? Ss 0:00 /usr/sbin/smbd -D 4109 ? S 0:00 /usr/sbin/smbd -D 13926 ? D< 0:00 [txg_sync] 15207 ? D 0:00 zfs snapshot -o com.sun:auto-snapshot-desc - -r homebackup@zfs-auto-snap_frequent-2012-12-03-2100 18054 pts/1 S+ 0:00 grep D

Dec 3 21:02:50 homerouter kernel: [ 3120.612085] INFO: task txg_sync:13926 blocked for more than 120 seconds. Dec 3 21:02:50 homerouter kernel: [ 3120.612134] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 21:02:50 homerouter kernel: [ 3120.612194] txg_sync D ffff88011bc13780 0 13926 2 0x00000000 Dec 3 21:02:50 homerouter kernel: [ 3120.612279] ffff8800d80c47b0 0000000000000046 0000000000000001 ffffffff8160d020 Dec 3 21:02:50 homerouter kernel: [ 3120.612413] 0000000000013780 ffff8800d646dfd8 ffff8800d646dfd8 ffff8800d80c47b0 Dec 3 21:02:50 homerouter kernel: [ 3120.612546] ffffffff810363d8 ffff88010becb530 0000000000000000 ffff88010ff13c40 Dec 3 21:02:50 homerouter kernel: [ 3120.612680] Call Trace: Dec 3 21:02:50 homerouter kernel: [ 3120.612722] [] ? should_resched+0x5/0x23 Dec 3 21:02:50 homerouter kernel: [ 3120.612768] [] ? rwsem_down_failed_common+0xe0/0x114 Dec 3 21:02:50 homerouter kernel: [ 3120.612816] [] ? call_rwsem_down_write_failed+0x13/0x20 Dec 3 21:02:50 homerouter kernel: [ 3120.612862] [] ? down_write+0x25/0x27 Dec 3 21:02:50 homerouter kernel: [ 3120.612937] [] ? zap_get_leaf_byblk+0x141/0x19c [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613004] [] ? zap_deref_leaf+0x5e/0x67 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613070] [] ? fzap_add_cd+0x45/0x101 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613135] [] ? zap_hashbits+0x5/0x14 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613200] [] ? zap_hash+0x9b/0xb1 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613264] [] ? zap_name_alloc+0xba/0xce [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613330] [] ? zap_add+0xcb/0x122 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613391] [] ? dsl_dataset_snapshot_sync+0x3cd/0x466 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613471] [] ? dsl_props_set_sync+0x194/0x1af [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613532] [] ? snapshot_sync+0x1d/0xb2 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613592] [] ? snapshot_check+0x16/0xa8 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613658] [] ? dsl_sync_task_group_sync+0x11a/0x198 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613737] [] ? dsl_pool_sync+0x1df/0x402 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613804] [] ? spa_sync+0x47a/0x836 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613851] [] ? ktime_get_ts+0x5c/0x82 Dec 3 21:02:50 homerouter kernel: [ 3120.613916] [] ? txg_sync_thread+0x2bd/0x49a [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.613984] [] ? txg_thread_wait.isra.2+0x23/0x23 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.614037] [] ? thread_generic_wrapper+0x6a/0x75 [spl] Dec 3 21:02:50 homerouter kernel: [ 3120.614087] [] ? thread_create+0x2be/0x2be [spl] Dec 3 21:02:50 homerouter kernel: [ 3120.614134] [] ? kthread+0x76/0x7e Dec 3 21:02:50 homerouter kernel: [ 3120.614177] [] ? kernel_thread_helper+0x4/0x10 Dec 3 21:02:50 homerouter kernel: [ 3120.614223] [] ? kthread_worker_fn+0x139/0x139 Dec 3 21:02:50 homerouter kernel: [ 3120.614269] [] ? gs_change+0x13/0x13 Dec 3 21:02:50 homerouter kernel: [ 3120.614317] INFO: task zfs:15207 blocked for more than 120 seconds. Dec 3 21:02:50 homerouter kernel: [ 3120.614361] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 21:02:50 homerouter kernel: [ 3120.614421] zfs D ffff88011bc93780 0 15207 14904 0x00000000 Dec 3 21:02:50 homerouter kernel: [ 3120.614504] ffff88010f82a240 0000000000000086 0000000000000001 ffff8800d80c47b0 Dec 3 21:02:50 homerouter kernel: [ 3120.614638] 0000000000013780 ffff8800865f7fd8 ffff8800865f7fd8 ffff88010f82a240 Dec 3 21:02:50 homerouter kernel: [ 3120.614771] 0000000000000202 ffffffff8134cb14 ffff880107793290 ffff880107793288 Dec 3 21:02:50 homerouter kernel: [ 3120.614905] Call Trace: Dec 3 21:02:50 homerouter kernel: [ 3120.614945] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 21:02:50 homerouter kernel: [ 3120.614995] [] ? cv_wait_common+0xcb/0x15e [spl] Dec 3 21:02:50 homerouter kernel: [ 3120.615043] [] ? add_wait_queue+0x3c/0x3c Dec 3 21:02:50 homerouter kernel: [ 3120.615109] [] ? txg_wait_synced+0x12b/0x168 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.615186] [] ? dsl_sync_task_group_wait+0x18e/0x1b8 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.615261] [] ? dmu_objset_snapshot+0x177/0x264 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.615326] [] ? zfs_ioc_snapshot+0x1a8/0x1d9 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.615390] [] ? zfsdev_ioctl+0x103/0x159 [zfs] Dec 3 21:02:50 homerouter kernel: [ 3120.615437] [] ? do_vfs_ioctl+0x459/0x49a Dec 3 21:02:50 homerouter kernel: [ 3120.615482] [] ? remove_vma+0x64/0x6b Dec 3 21:02:50 homerouter kernel: [ 3120.615527] [] ? do_munmap+0x2da/0x2f3 Dec 3 21:02:50 homerouter kernel: [ 3120.615571] [] ? sys_ioctl+0x4b/0x72 Dec 3 21:02:50 homerouter kernel: [ 3120.615615] [] ? system_call_fastpath+0x16/0x1b Dec 3 21:04:50 homerouter kernel: [ 3240.612089] INFO: task txg_sync:13926 blocked for more than 120 seconds. Dec 3 21:04:50 homerouter kernel: [ 3240.612137] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 21:04:50 homerouter kernel: [ 3240.612197] txg_sync D ffff88011bc13780 0 13926 2 0x00000000 Dec 3 21:04:50 homerouter kernel: [ 3240.612282] ffff8800d80c47b0 0000000000000046 0000000000000001 ffffffff8160d020 Dec 3 21:04:50 homerouter kernel: [ 3240.612416] 0000000000013780 ffff8800d646dfd8 ffff8800d646dfd8 ffff8800d80c47b0 Dec 3 21:04:50 homerouter kernel: [ 3240.612549] ffffffff810363d8 ffff88010becb530 0000000000000000 ffff88010ff13c40 Dec 3 21:04:50 homerouter kernel: [ 3240.612697] Call Trace: Dec 3 21:04:50 homerouter kernel: [ 3240.612739] [] ? should_resched+0x5/0x23 Dec 3 21:04:50 homerouter kernel: [ 3240.612785] [] ? rwsem_down_failed_common+0xe0/0x114 Dec 3 21:04:50 homerouter kernel: [ 3240.612833] [] ? call_rwsem_down_write_failed+0x13/0x20 Dec 3 21:04:50 homerouter kernel: [ 3240.612879] [] ? down_write+0x25/0x27 Dec 3 21:04:50 homerouter kernel: [ 3240.612956] [] ? zap_get_leaf_byblk+0x141/0x19c [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613023] [] ? zap_deref_leaf+0x5e/0x67 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613090] [] ? fzap_add_cd+0x45/0x101 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613155] [] ? zap_hashbits+0x5/0x14 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613220] [] ? zap_hash+0x9b/0xb1 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613285] [] ? zap_name_alloc+0xba/0xce [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613350] [] ? zap_add+0xcb/0x122 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613412] [] ? dsl_dataset_snapshot_sync+0x3cd/0x466 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613492] [] ? dsl_props_set_sync+0x194/0x1af [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613554] [] ? snapshot_sync+0x1d/0xb2 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613613] [] ? snapshot_check+0x16/0xa8 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613679] [] ? dsl_sync_task_group_sync+0x11a/0x198 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613758] [] ? dsl_pool_sync+0x1df/0x402 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613825] [] ? spa_sync+0x47a/0x836 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.613870] [] ? ktime_get_ts+0x5c/0x82 Dec 3 21:04:50 homerouter kernel: [ 3240.613936] [] ? txg_sync_thread+0x2bd/0x49a [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.614004] [] ? txg_thread_wait.isra.2+0x23/0x23 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.614056] [] ? thread_generic_wrapper+0x6a/0x75 [spl] Dec 3 21:04:50 homerouter kernel: [ 3240.614107] [] ? thread_create+0x2be/0x2be [spl] Dec 3 21:04:50 homerouter kernel: [ 3240.614153] [] ? kthread+0x76/0x7e Dec 3 21:04:50 homerouter kernel: [ 3240.614197] [] ? kernel_thread_helper+0x4/0x10 Dec 3 21:04:50 homerouter kernel: [ 3240.614242] [] ? kthread_worker_fn+0x139/0x139 Dec 3 21:04:50 homerouter kernel: [ 3240.614287] [] ? gs_change+0x13/0x13 Dec 3 21:04:50 homerouter kernel: [ 3240.614335] INFO: task zfs:15207 blocked for more than 120 seconds. Dec 3 21:04:50 homerouter kernel: [ 3240.614379] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 21:04:50 homerouter kernel: [ 3240.614439] zfs D ffff88011bc93780 0 15207 14904 0x00000000 Dec 3 21:04:50 homerouter kernel: [ 3240.614523] ffff88010f82a240 0000000000000086 0000000000000001 ffff8800d80c47b0 Dec 3 21:04:50 homerouter kernel: [ 3240.614657] 0000000000013780 ffff8800865f7fd8 ffff8800865f7fd8 ffff88010f82a240 Dec 3 21:04:50 homerouter kernel: [ 3240.614790] 0000000000000202 ffffffff8134cb14 ffff880107793290 ffff880107793288 Dec 3 21:04:50 homerouter kernel: [ 3240.614923] Call Trace: Dec 3 21:04:50 homerouter kernel: [ 3240.614961] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 21:04:50 homerouter kernel: [ 3240.615011] [] ? cv_wait_common+0xcb/0x15e [spl] Dec 3 21:04:50 homerouter kernel: [ 3240.615057] [] ? add_wait_queue+0x3c/0x3c Dec 3 21:04:50 homerouter kernel: [ 3240.615123] [] ? txg_wait_synced+0x12b/0x168 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.615190] [] ? dsl_sync_task_group_wait+0x18e/0x1b8 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.616916] [] ? dmu_objset_snapshot+0x177/0x264 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.616984] [] ? zfs_ioc_snapshot+0x1a8/0x1d9 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.617049] [] ? zfsdev_ioctl+0x103/0x159 [zfs] Dec 3 21:04:50 homerouter kernel: [ 3240.617098] [] ? do_vfs_ioctl+0x459/0x49a Dec 3 21:04:50 homerouter kernel: [ 3240.617144] [] ? remove_vma+0x64/0x6b Dec 3 21:04:50 homerouter kernel: [ 3240.617201] [] ? do_munmap+0x2da/0x2f3 Dec 3 21:04:50 homerouter kernel: [ 3240.617251] [] ? sys_ioctl+0x4b/0x72 Dec 3 21:04:50 homerouter kernel: [ 3240.617302] [] ? system_call_fastpath+0x16/0x1b Dec 3 21:06:50 homerouter kernel: [ 3360.616084] INFO: task txg_sync:13926 blocked for more than 120 seconds. Dec 3 21:06:50 homerouter kernel: [ 3360.616133] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 21:06:50 homerouter kernel: [ 3360.616193] txg_sync D ffff88011bc13780 0 13926 2 0x00000000 Dec 3 21:06:50 homerouter kernel: [ 3360.616278] ffff8800d80c47b0 0000000000000046 0000000000000001 ffffffff8160d020 Dec 3 21:06:50 homerouter kernel: [ 3360.616412] 0000000000013780 ffff8800d646dfd8 ffff8800d646dfd8 ffff8800d80c47b0 Dec 3 21:06:50 homerouter kernel: [ 3360.616546] ffffffff810363d8 ffff88010becb530 0000000000000000 ffff88010ff13c40 Dec 3 21:06:50 homerouter kernel: [ 3360.616680] Call Trace: Dec 3 21:06:50 homerouter kernel: [ 3360.616722] [] ? should_resched+0x5/0x23 Dec 3 21:06:50 homerouter kernel: [ 3360.616768] [] ? rwsem_down_failed_common+0xe0/0x114 Dec 3 21:06:50 homerouter kernel: [ 3360.616816] [] ? call_rwsem_down_write_failed+0x13/0x20 Dec 3 21:06:50 homerouter kernel: [ 3360.616862] [] ? down_write+0x25/0x27 Dec 3 21:06:50 homerouter kernel: [ 3360.616937] [] ? zap_get_leaf_byblk+0x141/0x19c [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617005] [] ? zap_deref_leaf+0x5e/0x67 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617072] [] ? fzap_add_cd+0x45/0x101 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617137] [] ? zap_hashbits+0x5/0x14 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617202] [] ? zap_hash+0x9b/0xb1 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617266] [] ? zap_name_alloc+0xba/0xce [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617331] [] ? zap_add+0xcb/0x122 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617393] [] ? dsl_dataset_snapshot_sync+0x3cd/0x466 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617473] [] ? dsl_props_set_sync+0x194/0x1af [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617535] [] ? snapshot_sync+0x1d/0xb2 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617595] [] ? snapshot_check+0x16/0xa8 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617660] [] ? dsl_sync_task_group_sync+0x11a/0x198 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617740] [] ? dsl_pool_sync+0x1df/0x402 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617807] [] ? spa_sync+0x47a/0x836 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617852] [] ? ktime_get_ts+0x5c/0x82 Dec 3 21:06:50 homerouter kernel: [ 3360.617917] [] ? txg_sync_thread+0x2bd/0x49a [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.617985] [] ? txg_thread_wait.isra.2+0x23/0x23 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.618037] [] ? thread_generic_wrapper+0x6a/0x75 [spl] Dec 3 21:06:50 homerouter kernel: [ 3360.618087] [] ? thread_create+0x2be/0x2be [spl] Dec 3 21:06:50 homerouter kernel: [ 3360.618134] [] ? kthread+0x76/0x7e Dec 3 21:06:50 homerouter kernel: [ 3360.618178] [] ? kernel_thread_helper+0x4/0x10 Dec 3 21:06:50 homerouter kernel: [ 3360.618223] [] ? kthread_worker_fn+0x139/0x139 Dec 3 21:06:50 homerouter kernel: [ 3360.618268] [] ? gs_change+0x13/0x13 Dec 3 21:06:50 homerouter kernel: [ 3360.618315] INFO: task zfs:15207 blocked for more than 120 seconds. Dec 3 21:06:50 homerouter kernel: [ 3360.618361] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 21:06:50 homerouter kernel: [ 3360.618422] zfs D ffff88011bc93780 0 15207 14904 0x00000000 Dec 3 21:06:50 homerouter kernel: [ 3360.618521] ffff88010f82a240 0000000000000086 0000000000000001 ffff8800d80c47b0 Dec 3 21:06:50 homerouter kernel: [ 3360.618663] 0000000000013780 ffff8800865f7fd8 ffff8800865f7fd8 ffff88010f82a240 Dec 3 21:06:50 homerouter kernel: [ 3360.618797] 0000000000000202 ffffffff8134cb14 ffff880107793290 ffff880107793288 Dec 3 21:06:50 homerouter kernel: [ 3360.618931] Call Trace: Dec 3 21:06:50 homerouter kernel: [ 3360.618972] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 21:06:50 homerouter kernel: [ 3360.619026] [] ? cv_wait_common+0xcb/0x15e [spl] Dec 3 21:06:50 homerouter kernel: [ 3360.619072] [] ? add_wait_queue+0x3c/0x3c Dec 3 21:06:50 homerouter kernel: [ 3360.619138] [] ? txg_wait_synced+0x12b/0x168 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.619205] [] ? dsl_sync_task_group_wait+0x18e/0x1b8 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.619280] [] ? dmu_objset_snapshot+0x177/0x264 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.619345] [] ? zfs_ioc_snapshot+0x1a8/0x1d9 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.619410] [] ? zfsdev_ioctl+0x103/0x159 [zfs] Dec 3 21:06:50 homerouter kernel: [ 3360.619457] [] ? do_vfs_ioctl+0x459/0x49a Dec 3 21:06:50 homerouter kernel: [ 3360.619503] [] ? remove_vma+0x64/0x6b Dec 3 21:06:50 homerouter kernel: [ 3360.619546] [] ? do_munmap+0x2da/0x2f3 Dec 3 21:06:50 homerouter kernel: [ 3360.619590] [] ? sys_ioctl+0x4b/0x72 Dec 3 21:06:50 homerouter kernel: [ 3360.619635] [] ? system_call_fastpath+0x16/0x1b Dec 3 21:08:50 homerouter kernel: [ 3480.616112] INFO: task txg_sync:13926 blocked for more than 120 seconds. Dec 3 21:08:50 homerouter kernel: [ 3480.616162] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 21:08:50 homerouter kernel: [ 3480.616224] txg_sync D ffff88011bc13780 0 13926 2 0x00000000 Dec 3 21:08:50 homerouter kernel: [ 3480.616311] ffff8800d80c47b0 0000000000000046 0000000000000001 ffffffff8160d020 Dec 3 21:08:50 homerouter kernel: [ 3480.616496] 0000000000013780 ffff8800d646dfd8 ffff8800d646dfd8 ffff8800d80c47b0 Dec 3 21:08:50 homerouter kernel: [ 3480.616643] ffffffff810363d8 ffff88010becb530 0000000000000000 ffff88010ff13c40 Dec 3 21:08:50 homerouter kernel: [ 3480.616779] Call Trace: Dec 3 21:08:50 homerouter kernel: [ 3480.616822] [] ? should_resched+0x5/0x23 Dec 3 21:08:50 homerouter kernel: [ 3480.616869] [] ? rwsem_down_failed_common+0xe0/0x114 Dec 3 21:08:50 homerouter kernel: [ 3480.616918] [] ? call_rwsem_down_write_failed+0x13/0x20 Dec 3 21:08:50 homerouter kernel: [ 3480.616967] [] ? down_write+0x25/0x27 Dec 3 21:08:50 homerouter kernel: [ 3480.617045] [] ? zap_get_leaf_byblk+0x141/0x19c [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617114] [] ? zap_deref_leaf+0x5e/0x67 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617182] [] ? fzap_add_cd+0x45/0x101 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617250] [] ? zap_hashbits+0x5/0x14 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617317] [] ? zap_hash+0x9b/0xb1 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617391] [] ? zap_name_alloc+0xba/0xce [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617460] [] ? zap_add+0xcb/0x122 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617522] [] ? dsl_dataset_snapshot_sync+0x3cd/0x466 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617604] [] ? dsl_props_set_sync+0x194/0x1af [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617665] [] ? snapshot_sync+0x1d/0xb2 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617725] [] ? snapshot_check+0x16/0xa8 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617791] [] ? dsl_sync_task_group_sync+0x11a/0x198 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617871] [] ? dsl_pool_sync+0x1df/0x402 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617939] [] ? spa_sync+0x47a/0x836 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.617985] [] ? ktime_get_ts+0x5c/0x82 Dec 3 21:08:50 homerouter kernel: [ 3480.618050] [] ? txg_sync_thread+0x2bd/0x49a [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.618119] [] ? txg_thread_wait.isra.2+0x23/0x23 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.618171] [] ? thread_generic_wrapper+0x6a/0x75 [spl] Dec 3 21:08:50 homerouter kernel: [ 3480.618222] [] ? thread_create+0x2be/0x2be [spl] Dec 3 21:08:50 homerouter kernel: [ 3480.618269] [] ? kthread+0x76/0x7e Dec 3 21:08:50 homerouter kernel: [ 3480.618314] [] ? kernel_thread_helper+0x4/0x10 Dec 3 21:08:50 homerouter kernel: [ 3480.618360] [] ? kthread_worker_fn+0x139/0x139 Dec 3 21:08:50 homerouter kernel: [ 3480.618405] [] ? gs_change+0x13/0x13 Dec 3 21:08:50 homerouter kernel: [ 3480.618454] INFO: task zfs:15207 blocked for more than 120 seconds. Dec 3 21:08:50 homerouter kernel: [ 3480.618498] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Dec 3 21:08:50 homerouter kernel: [ 3480.618559] zfs D ffff88011bc93780 0 15207 14904 0x00000000 Dec 3 21:08:50 homerouter kernel: [ 3480.618644] ffff88010f82a240 0000000000000086 0000000000000001 ffff8800d80c47b0 Dec 3 21:08:50 homerouter kernel: [ 3480.618779] 0000000000013780 ffff8800865f7fd8 ffff8800865f7fd8 ffff88010f82a240 Dec 3 21:08:50 homerouter kernel: [ 3480.618915] 0000000000000202 ffffffff8134cb14 ffff880107793290 ffff880107793288 Dec 3 21:08:50 homerouter kernel: [ 3480.619050] Call Trace: Dec 3 21:08:50 homerouter kernel: [ 3480.619089] [] ? _raw_spin_lock_irqsave+0x9/0x25 Dec 3 21:08:50 homerouter kernel: [ 3480.619140] [] ? cv_wait_common+0xcb/0x15e [spl] Dec 3 21:08:50 homerouter kernel: [ 3480.619186] [] ? add_wait_queue+0x3c/0x3c Dec 3 21:08:50 homerouter kernel: [ 3480.619253] [] ? txg_wait_synced+0x12b/0x168 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.619321] [] ? dsl_sync_task_group_wait+0x18e/0x1b8 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.619397] [] ? dmu_objset_snapshot+0x177/0x264 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.619462] [] ? zfs_ioc_snapshot+0x1a8/0x1d9 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.619527] [] ? zfsdev_ioctl+0x103/0x159 [zfs] Dec 3 21:08:50 homerouter kernel: [ 3480.619575] [] ? do_vfs_ioctl+0x459/0x49a Dec 3 21:08:50 homerouter kernel: [ 3480.619620] [] ? remove_vma+0x64/0x6b Dec 3 21:08:50 homerouter kernel: [ 3480.619664] [] ? do_munmap+0x2da/0x2f3 Dec 3 21:08:50 homerouter kernel: [ 3480.619709] [] ? sys_ioctl+0x4b/0x72 Dec 3 21:08:50 homerouter kernel: [ 3480.619754] [] ? system_call_fastpath+0x16/0x1b

[behlendorf]

zfsonlinux/zfs@1305d33 fzap_cursor_move_to_key() should drop l_rwlock

[iamacarpet]

Is anyone else still having this issue? I'm using the Ubuntu PPA for the RCs and I recently updated to RC14 and still seem to be experiencing this exact same issue.

Cron is running zfs-auto-snapshot and if two of them run at the same time because the disk is otherwise busy, the whole filesystem hangs (the filesystem was busy because I was running a scrub to make sure no damage was present from the issue happening on the previous RC).

Feb 6 23:21:12 santos kernel: [44521.368218] INFO: task txg_sync:522 blocked for more than 120 seconds. Feb 6 23:21:12 santos kernel: [44521.368279] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 6 23:21:12 santos kernel: [44521.368333] txg_sync D 0000000000000001 0 522 2 0x00000000 Feb 6 23:21:12 santos kernel: [44521.368344] ffff880075155b40 0000000000000046 ffff880075155af0 ffff88007943af98 Feb 6 23:21:12 santos kernel: [44521.368356] ffff880075155fd8 ffff880075155fd8 ffff880075155fd8 00000000000137c0 Feb 6 23:21:12 santos kernel: [44521.368366] ffff880076ee5c00 ffff880079660000 ffff880075155b30 ffff88007dc94080 Feb 6 23:21:12 santos kernel: [44521.368375] Call Trace: Feb 6 23:21:12 santos kernel: [44521.368390] [] schedule+0x3f/0x60 Feb 6 23:21:12 santos kernel: [44521.368398] [] io_schedule+0x8f/0xd0 Feb 6 23:21:12 santos kernel: [44521.368441] [] cv_wait_common+0xa8/0x1b0 [spl] Feb 6 23:21:12 santos kernel: [44521.368452] [] ? add_wait_queue+0x60/0x60 Feb 6 23:21:12 santos kernel: [44521.368477] [] cv_wait_io+0x18/0x20 [spl] Feb 6 23:21:12 santos kernel: [44521.368567] [] zio_wait+0x103/0x1a0 [zfs] Feb 6 23:21:12 santos kernel: [44521.368641] [] dsl_scan_sync+0x488/0xaa0 [zfs] Feb 6 23:21:12 santos kernel: [44521.368719] [] spa_sync+0x40b/0xa50 [zfs] Feb 6 23:21:12 santos kernel: [44521.368798] [] txg_sync_thread+0x2df/0x540 [zfs] Feb 6 23:21:12 santos kernel: [44521.368879] [] ? txg_init+0x250/0x250 [zfs] Feb 6 23:21:12 santos kernel: [44521.368902] [] thread_generic_wrapper+0x78/0x90 [spl] Feb 6 23:21:12 santos kernel: [44521.368924] [] ? __thread_create+0x310/0x310 [spl] Feb 6 23:21:12 santos kernel: [44521.368932] [] kthread+0x8c/0xa0 Feb 6 23:21:12 santos kernel: [44521.368942] [] kernel_thread_helper+0x4/0x10 Feb 6 23:21:12 santos kernel: [44521.368950] [] ? flush_kthread_worker+0xa0/0xa0 Feb 6 23:21:12 santos kernel: [44521.368958] [] ? gs_change+0x13/0x13 Feb 6 23:21:12 santos kernel: [44521.369004] INFO: task zfs:10515 blocked for more than 120 seconds. Feb 6 23:21:12 santos kernel: [44521.369047] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Feb 6 23:21:12 santos kernel: [44521.369098] zfs D 0000000000000000 0 10515 10514 0x00000000 Feb 6 23:21:12 santos kernel: [44521.369108] ffff8800044dfc88 0000000000000082 ffff8800044dfd28 ffffffffa00b5973 Feb 6 23:21:12 santos kernel: [44521.369118] ffff8800044dffd8 ffff8800044dffd8 ffff8800044dffd8 00000000000137c0 Feb 6 23:21:12 santos kernel: [44521.369127] ffff880079661700 ffff880079ffc500 ffff8800044dfc98 ffff880078fa2a90 Feb 6 23:21:12 santos kernel: [44521.369136] Call Trace: Feb 6 23:21:12 santos kernel: [44521.369157] [] ? spl_debug_msg+0x483/0x8e0 [spl] Feb 6 23:21:12 santos kernel: [44521.369165] [] schedule+0x3f/0x60 Feb 6 23:21:12 santos kernel: [44521.369188] [] cv_wait_common+0xfd/0x1b0 [spl] Feb 6 23:21:12 santos kernel: [44521.369197] [] ? add_wait_queue+0x60/0x60 Feb 6 23:21:12 santos kernel: [44521.369219] [] cv_wait+0x15/0x20 [spl] Feb 6 23:21:12 santos kernel: [44521.369299] [] txg_wait_synced+0xb3/0x190 [zfs] Feb 6 23:21:12 santos kernel: [44521.369375] [] dsl_sync_task_group_wait+0x17d/0x2a0 [zfs] Feb 6 23:21:12 santos kernel: [44521.369446] [] dmu_snapshots_destroy_nvl+0x128/0x1f0 [zfs] Feb 6 23:21:12 santos kernel: [44521.369523] [] ? zfs_unmount_snap+0x92/0x130 [zfs] Feb 6 23:21:12 santos kernel: [44521.369600] [] zfs_ioc_destroy_snaps_nvl+0xfa/0x130 [zfs] Feb 6 23:21:12 santos kernel: [44521.369677] [] ? spa_open+0x13/0x20 [zfs] Feb 6 23:21:12 santos kernel: [44521.369755] [] zfsdev_ioctl+0xdc/0x1b0 [zfs] Feb 6 23:21:12 santos kernel: [44521.369765] [] do_vfs_ioctl+0x8a/0x340 Feb 6 23:21:12 santos kernel: [44521.369774] [] ? __schedule+0x3cc/0x6f0 Feb 6 23:21:12 santos kernel: [44521.369783] [] ? do_munmap+0x1f3/0x2f0 Feb 6 23:21:12 santos kernel: [44521.369791] [] sys_ioctl+0x91/0xa0 Feb 6 23:21:12 santos kernel: [44521.369799] [] system_call_fastpath+0x16/0x1b

[behlendorf]

@ iamacarpet This looks like a different issue to me. Your system is blocked waiting on an outstanding I/O to the disk for some unknown reason. Check zpool status and zpool events, there may be additional information in there about the delayed I/Os.

[iamacarpet]

Thanks, I've opened a new ticket about it here with a little more info: https://github.com/zfsonlinux/zfs/issues/1283