search

openzfs / zfs

OpenZFS on Linux and FreeBSD
https://openzfs.github.io/openzfs-docs
Other
10.61k stars 1.75k forks source link

BackupPC processes hang after upgrade to 0.6.4 #3302

Closed rgmiller closed 9 years ago

rgmiller commented 9 years ago

I've got a server running CentOS 7 with 2 zpools. One of these pools is the target for BackupPC. Since upgrading from 0.6.3 to 0.6.4 (using the packages from the ZoL repo) my BackupPC jobs have starting hanging. The stack traces from the syslog look like some kind of deadlock in the zfs code to me. (I'll upload the stack traces as soon as I figure out how to do that with GitHub.)

A few other potentially relevant details: 1) Everything was fine under 0.6.3 (which is one of the reasons why I think this is a ZFS problem, not a BackupPC problem) 2) BackupPC is the only application accessing the zpool, but it does have multiple processes running simultaneously 3) Whether or not the process hangs seems to be related to the amount of data written (Incremental backups will sometimes complete successfully. No full backup has completed since I upgraded to 0.6.4.) 4) I have not enabled any of the new features in 0.6.4 on either of the zpools.

rgmiller commented 9 years ago

Stack trace from ~2AM Tuesday morning (the first time it happened):

Apr 14 02:03:37 opus kernel: INFO: task txg_quiesce:1305 blocked for more than 120 seconds. Apr 14 02:03:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:03:37 opus kernel: txg_quiesce D ffff88041e393680 0 1305 2 0x00000000 Apr 14 02:03:37 opus kernel: ffff8803f7f67d40 0000000000000046 ffff8803f7f67fd8 0000000000013680 Apr 14 02:03:37 opus kernel: ffff8803f7f67fd8 0000000000013680 ffff8803feb2c440 ffff8804071bd7f0 Apr 14 02:03:37 opus kernel: ffff8804071bd6e0 ffff8804071bd7f8 ffff8804071bd708 0000000000000000 Apr 14 02:03:37 opus kernel: Call Trace: Apr 14 02:03:37 opus kernel: [] schedule+0x29/0x70 Apr 14 02:03:37 opus kernel: [] cv_wait_common+0x125/0x150 [spl] Apr 14 02:03:37 opus kernel: [] ? wake_up_bit+0x30/0x30 Apr 14 02:03:37 opus kernel: [] cv_wait+0x15/0x20 [spl] Apr 14 02:03:37 opus kernel: [] txg_quiesce_thread+0x2bb/0x3a0 [zfs] Apr 14 02:03:37 opus kernel: [] ? txg_init+0x2b0/0x2b0 [zfs] Apr 14 02:03:37 opus kernel: [] thread_generic_wrapper+0x71/0x80 [spl] Apr 14 02:03:37 opus kernel: [] ? thread_exit+0x20/0x20 [spl] Apr 14 02:03:37 opus kernel: [] kthread+0xcf/0xe0 Apr 14 02:03:37 opus kernel: [] ? kthread_create_on_node+0x140/0x140 Apr 14 02:03:37 opus kernel: [] ret_from_fork+0x7c/0xb0 Apr 14 02:03:37 opus kernel: [] ? kthread_create_on_node+0x140/0x140 Apr 14 02:03:37 opus kernel: INFO: task BackupPC_nightl:5468 blocked for more than 120 seconds. Apr 14 02:03:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:03:37 opus kernel: BackupPC_nightl D ffff88041e393680 0 5468 3731 0x00000080 Apr 14 02:03:37 opus kernel: ffff88033b3ab940 0000000000000082 ffff88033b3abfd8 0000000000013680 Apr 14 02:03:37 opus kernel: ffff88033b3abfd8 0000000000013680 ffff8803f0638b60 ffff8803f4aadb58 Apr 14 02:03:37 opus kernel: ffff8803f4aadb5c ffff8803f0638b60 00000000ffffffff ffff8803f4aadb60 Apr 14 02:03:37 opus kernel: Call Trace: Apr 14 02:03:37 opus kernel: [] schedule_preempt_disabled+0x29/0x70 Apr 14 02:03:37 opus kernel: [] mutex_lock_slowpath+0xc5/0x1c0 Apr 14 02:03:37 opus kernel: [] mutex_lock+0x1f/0x2f Apr 14 02:03:37 opus kernel: [] zfs_zget+0x176/0x310 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_dirent_lock+0x486/0x5c0 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_dirlook+0x90/0x380 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_lookup+0x306/0x360 [zfs] Apr 14 02:03:37 opus kernel: [] zpl_lookup+0xae/0x130 [zfs] Apr 14 02:03:37 opus kernel: [] lookup_real+0x1d/0x50 Apr 14 02:03:37 opus kernel: [] lookup_hash+0x42/0x60 Apr 14 02:03:37 opus kernel: [] lookup_slow+0x42/0xa7 Apr 14 02:03:37 opus kernel: [] path_lookupat+0x773/0x7a0 Apr 14 02:03:37 opus kernel: [] ? kmem_cache_alloc+0x35/0x1d0 Apr 14 02:03:37 opus kernel: [] ? getname_flags+0x4f/0x190 Apr 14 02:03:37 opus kernel: [] filename_lookup+0x2b/0xc0 Apr 14 02:03:37 opus kernel: [] user_path_at_empty+0x67/0xc0 Apr 14 02:03:37 opus kernel: [] ? from_kgid_munged+0x12/0x20 Apr 14 02:03:37 opus kernel: [] ? cp_new_stat+0x14f/0x180 Apr 14 02:03:37 opus kernel: [] user_path_at+0x11/0x20 Apr 14 02:03:37 opus kernel: [] vfs_fstatat+0x63/0xc0 Apr 14 02:03:37 opus kernel: [] SYSC_newstat+0x2e/0x60 Apr 14 02:03:37 opus kernel: [] ? audit_syscall_exit+0x1f6/0x2a0 Apr 14 02:03:37 opus kernel: [] SyS_newstat+0xe/0x10 Apr 14 02:03:37 opus kernel: [] system_call_fastpath+0x16/0x1b Apr 14 02:03:37 opus kernel: INFO: task BackupPC_nightl:5469 blocked for more than 120 seconds. Apr 14 02:03:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:03:37 opus kernel: BackupPC_nightl D ffff88041e313680 0 5469 3731 0x00000080 Apr 14 02:03:37 opus kernel: ffff8803b099ef00 0000000000000086 ffff8803b099ffd8 0000000000013680 Apr 14 02:03:37 opus kernel: ffff8803b099ffd8 0000000000013680 ffff8803f0638000 ffff8803f4aadb58 Apr 14 02:03:37 opus kernel: ffff8803f4aadb5c ffff8803f0638000 00000000ffffffff ffff8803f4aadb60 Apr 14 02:03:37 opus kernel: Call Trace: Apr 14 02:03:37 opus kernel: [] schedule_preempt_disabled+0x29/0x70 Apr 14 02:03:37 opus kernel: [] mutex_lock_slowpath+0xc5/0x1c0 Apr 14 02:03:37 opus kernel: [] mutex_lock+0x1f/0x2f Apr 14 02:03:37 opus kernel: [] zfs_zinactive+0x52/0x180 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_inactive+0x64/0x230 [zfs] Apr 14 02:03:37 opus kernel: [] zpl_evict_inode+0x43/0x60 [zfs] Apr 14 02:03:37 opus kernel: [] evict+0xa7/0x170 Apr 14 02:03:37 opus kernel: [] dispose_list+0x3e/0x50 Apr 14 02:03:37 opus kernel: [] prune_icache_sb+0x163/0x320 Apr 14 02:03:37 opus kernel: [] prune_super+0xd6/0x1a0 Apr 14 02:03:37 opus kernel: [] shrink_slab+0x165/0x300 Apr 14 02:03:37 opus kernel: [] ? vmpressure+0x21/0x90 Apr 14 02:03:37 opus kernel: [] do_try_to_free_pages+0x3c2/0x4e0 Apr 14 02:03:37 opus kernel: [] try_to_free_pages+0xfc/0x180 Apr 14 02:03:37 opus kernel: [] alloc_pages_nodemask+0x7fd/0xb90 Apr 14 02:03:37 opus kernel: [] alloc_pages_current+0xa9/0x170 Apr 14 02:03:37 opus kernel: [] new_slab+0x275/0x300 Apr 14 02:03:37 opus kernel: [] slab_alloc+0x315/0x48f Apr 14 02:03:37 opus kernel: [] ? spl_kmem_cache_alloc+0xaa/0x180 [spl] Apr 14 02:03:37 opus kernel: [] kmem_cache_alloc+0x193/0x1d0 Apr 14 02:03:37 opus kernel: [] ? spl_kmem_cache_alloc+0xaa/0x180 [spl] Apr 14 02:03:37 opus kernel: [] spl_kmem_cache_alloc+0xaa/0x180 [spl] Apr 14 02:03:37 opus kernel: [] zio_create+0x3d/0x500 [zfs] Apr 14 02:03:37 opus kernel: [] zio_null+0x61/0x70 [zfs] Apr 14 02:03:37 opus kernel: [] zio_root+0x1e/0x20 [zfs] Apr 14 02:03:37 opus kernel: [] dbuf_read+0x6e6/0xa10 [zfs] Apr 14 02:03:37 opus kernel: [] ? dbuf_create+0x467/0x500 [zfs] Apr 14 02:03:37 opus kernel: [] ? dbuf_rele+0x36/0x40 [zfs] Apr 14 02:03:37 opus kernel: [] ? dnode_rele_and_unlock+0x5b/0x90 [zfs] Apr 14 02:03:37 opus kernel: [] dmu_bonus_hold+0xfc/0x300 [zfs] Apr 14 02:03:37 opus kernel: [] sa_buf_hold+0xe/0x10 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_zget+0x19c/0x310 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_dirent_lock+0x486/0x5c0 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_dirlook+0x90/0x380 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_lookup+0x306/0x360 [zfs] Apr 14 02:03:37 opus kernel: [] zpl_xattr_get+0x108/0x2a0 [zfs] Apr 14 02:03:37 opus kernel: [] zpl_xattr_get+0x5c/0x120 [zfs] Apr 14 02:03:37 opus kernel: [] zpl_xattr_security_get+0x50/0x70 [zfs] Apr 14 02:03:37 opus kernel: [] generic_getxattr+0x4c/0x70 Apr 14 02:03:37 opus kernel: [] inode_doinit_with_dentry+0x16f/0x680 Apr 14 02:03:37 opus kernel: [] ? tsd_exit+0x1fb/0x210 [spl] Apr 14 02:03:37 opus kernel: [] selinux_d_instantiate+0x1c/0x20 Apr 14 02:03:37 opus kernel: [] security_d_instantiate+0x1b/0x30 Apr 14 02:03:37 opus kernel: [] d_instantiate+0x50/0x70 Apr 14 02:03:37 opus kernel: [] d_splice_alias+0x44/0x120 Apr 14 02:03:37 opus kernel: [] zpl_lookup+0x11c/0x130 [zfs] Apr 14 02:03:37 opus kernel: [] lookup_real+0x1d/0x50 Apr 14 02:03:37 opus kernel: [] lookup_hash+0x42/0x60 Apr 14 02:03:37 opus kernel: [] lookup_slow+0x42/0xa7 Apr 14 02:03:37 opus kernel: [] path_lookupat+0x773/0x7a0 Apr 14 02:03:37 opus kernel: [] ? kmem_cache_alloc+0x35/0x1d0 Apr 14 02:03:37 opus kernel: [] ? getname_flags+0x4f/0x190 Apr 14 02:03:37 opus kernel: [] filename_lookup+0x2b/0xc0 Apr 14 02:03:37 opus kernel: [] user_path_at_empty+0x67/0xc0 Apr 14 02:03:37 opus kernel: [] ? from_kgid_munged+0x12/0x20 Apr 14 02:03:37 opus kernel: [] ? cp_new_stat+0x14f/0x180 Apr 14 02:03:37 opus kernel: [] user_path_at+0x11/0x20 Apr 14 02:03:37 opus kernel: [] vfs_fstatat+0x63/0xc0 Apr 14 02:03:37 opus kernel: [] SYSC_newstat+0x2e/0x60 Apr 14 02:03:37 opus kernel: [] ? audit_syscall_exit+0x1f6/0x2a0 Apr 14 02:03:37 opus kernel: [] SyS_newstat+0xe/0x10 Apr 14 02:03:37 opus kernel: [] system_call_fastpath+0x16/0x1b Apr 14 02:03:37 opus kernel: INFO: task BackupPC_dump:5722 blocked for more than 120 seconds. Apr 14 02:03:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:03:37 opus kernel: BackupPC_dump D ffff88041e213680 0 5722 3731 0x00000080 Apr 14 02:03:37 opus kernel: ffff8802086cb9b0 0000000000000086 ffff8802086cbfd8 0000000000013680 Apr 14 02:03:37 opus kernel: ffff8802086cbfd8 0000000000013680 ffff8804080f16c0 ffff8803f4aadb58 Apr 14 02:03:37 opus kernel: ffff8803f4aadb5c ffff8804080f16c0 00000000ffffffff ffff8803f4aadb60 Apr 14 02:03:37 opus kernel: Call Trace: Apr 14 02:03:37 opus kernel: [] schedule_preempt_disabled+0x29/0x70 Apr 14 02:03:37 opus kernel: [] mutex_lock_slowpath+0xc5/0x1c0 Apr 14 02:03:37 opus kernel: [] mutex_lock+0x1f/0x2f Apr 14 02:03:37 opus kernel: [] zfs_zget+0x176/0x310 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_dirent_lock+0x486/0x5c0 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_dirlook+0x90/0x380 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_lookup+0x306/0x360 [zfs] Apr 14 02:03:37 opus kernel: [] zpl_lookup+0xae/0x130 [zfs] Apr 14 02:03:37 opus kernel: [] lookup_real+0x1d/0x50 Apr 14 02:03:37 opus kernel: [] do_last+0xb23/0x1270 Apr 14 02:03:37 opus kernel: [] ? zfs_dirty_inode+0x2b1/0x330 [zfs] Apr 14 02:03:37 opus kernel: [] ? kmem_cache_alloc_trace+0x1ce/0x1f0 Apr 14 02:03:37 opus kernel: [] path_openat+0xc2/0x4c0 Apr 14 02:03:37 opus kernel: [] do_filp_open+0x4b/0xb0 Apr 14 02:03:37 opus kernel: [] ? alloc_fd+0xa7/0x130 Apr 14 02:03:37 opus kernel: [] do_sys_open+0xf3/0x1f0 Apr 14 02:03:37 opus kernel: [] SyS_open+0x1e/0x20 Apr 14 02:03:37 opus kernel: [] system_call_fastpath+0x16/0x1b Apr 14 02:03:37 opus kernel: INFO: task BackupPC_dump:5723 blocked for more than 120 seconds. Apr 14 02:03:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:03:37 opus kernel: BackupPC_dump D ffff88041e213680 0 5723 3731 0x00000080 Apr 14 02:03:37 opus kernel: ffff8802086939b0 0000000000000082 ffff880208693fd8 0000000000013680 Apr 14 02:03:37 opus kernel: ffff880208693fd8 0000000000013680 ffff8804055ae660 ffff8803f4aadb58 Apr 14 02:03:37 opus kernel: ffff8803f4aadb5c ffff8804055ae660 00000000ffffffff ffff8803f4aadb60 Apr 14 02:03:37 opus kernel: Call Trace: Apr 14 02:03:37 opus kernel: [] schedule_preempt_disabled+0x29/0x70 Apr 14 02:03:37 opus kernel: [] mutex_lock_slowpath+0xc5/0x1c0 Apr 14 02:03:37 opus kernel: [] mutex_lock+0x1f/0x2f Apr 14 02:03:37 opus kernel: [] zfs_zget+0x176/0x310 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_dirent_lock+0x486/0x5c0 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_dirlook+0x90/0x380 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_lookup+0x306/0x360 [zfs] Apr 14 02:03:37 opus kernel: [] zpl_lookup+0xae/0x130 [zfs] Apr 14 02:03:37 opus kernel: [] lookup_real+0x1d/0x50 Apr 14 02:03:37 opus kernel: [] do_last+0xb23/0x1270 Apr 14 02:03:37 opus kernel: [] ? kmem_cache_alloc_trace+0x1ce/0x1f0 Apr 14 02:03:37 opus kernel: [] path_openat+0xc2/0x4c0 Apr 14 02:03:37 opus kernel: [] ? tsd_hash_search.isra.0+0xb8/0xd0 [spl] Apr 14 02:03:37 opus kernel: [] ? tsd_exit+0x1fb/0x210 [spl] Apr 14 02:03:37 opus kernel: [] do_filp_open+0x4b/0xb0 Apr 14 02:03:37 opus kernel: [] ? alloc_fd+0xa7/0x130 Apr 14 02:03:37 opus kernel: [] do_sys_open+0xf3/0x1f0 Apr 14 02:03:37 opus kernel: [] SyS_openat+0x14/0x20 Apr 14 02:03:37 opus kernel: [] system_call_fastpath+0x16/0x1b Apr 14 02:03:37 opus kernel: INFO: task sshd:5772 blocked for more than 120 seconds. Apr 14 02:03:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:03:37 opus kernel: sshd D ffff88041e393680 0 5772 2191 0x00000080 Apr 14 02:03:37 opus kernel: ffff880209b2fd00 0000000000000086 ffff880209b2ffd8 0000000000013680 Apr 14 02:03:37 opus kernel: ffff880209b2ffd8 0000000000013680 ffff8803eda7a220 ffff8802d06f1e40 Apr 14 02:03:37 opus kernel: ffff8802d06f1e44 ffff8803eda7a220 00000000ffffffff ffff8802d06f1e48 Apr 14 02:03:37 opus kernel: Call Trace: Apr 14 02:03:37 opus kernel: [] schedule_preempt_disabled+0x29/0x70 Apr 14 02:03:37 opus kernel: [] mutex_lock_slowpath+0xc5/0x1c0 Apr 14 02:03:37 opus kernel: [] mutex_lock+0x1f/0x2f Apr 14 02:03:37 opus kernel: [] pipe_read+0x68/0x4e0 Apr 14 02:03:37 opus kernel: [] do_sync_read+0x8d/0xd0 Apr 14 02:03:37 opus kernel: [] vfs_read+0x9c/0x170 Apr 14 02:03:37 opus kernel: [] SyS_read+0x58/0xb0 Apr 14 02:03:37 opus kernel: [] system_call_fastpath+0x16/0x1b Apr 14 02:03:37 opus kernel: INFO: task gtar:5775 blocked for more than 120 seconds. Apr 14 02:03:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:03:37 opus kernel: gtar D ffff88041e293680 0 5775 5772 0x00000080 Apr 14 02:03:37 opus kernel: ffff880205d57840 0000000000000082 ffff880205d57fd8 0000000000013680 Apr 14 02:03:37 opus kernel: ffff880205d57fd8 0000000000013680 ffff8803ede12d80 ffff8803f4aadb58 Apr 14 02:03:37 opus kernel: ffff8803f4aadb5c ffff8803ede12d80 00000000ffffffff ffff8803f4aadb60 Apr 14 02:03:37 opus kernel: Call Trace: Apr 14 02:03:37 opus kernel: [] schedule_preempt_disabled+0x29/0x70 Apr 14 02:03:37 opus kernel: [] mutex_lock_slowpath+0xc5/0x1c0 Apr 14 02:03:37 opus kernel: [] mutex_lock+0x1f/0x2f Apr 14 02:03:37 opus kernel: [] zfs_zinactive+0x52/0x180 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_inactive+0x64/0x230 [zfs] Apr 14 02:03:37 opus kernel: [] zpl_evict_inode+0x43/0x60 [zfs] Apr 14 02:03:37 opus kernel: [] evict+0xa7/0x170 Apr 14 02:03:37 opus kernel: [] dispose_list+0x3e/0x50 Apr 14 02:03:37 opus kernel: [] prune_icache_sb+0x163/0x320 Apr 14 02:03:37 opus kernel: [] prune_super+0xd6/0x1a0 Apr 14 02:03:37 opus kernel: [] shrink_slab+0x165/0x300 Apr 14 02:03:37 opus kernel: [] ? vmpressure+0x21/0x90 Apr 14 02:03:37 opus kernel: [] do_try_to_free_pages+0x3c2/0x4e0 Apr 14 02:03:37 opus kernel: [] try_to_free_pages+0xfc/0x180 Apr 14 02:03:37 opus kernel: [] alloc_pages_nodemask+0x7fd/0xb90 Apr 14 02:03:37 opus kernel: [] alloc_pages_current+0xa9/0x170 Apr 14 02:03:37 opus kernel: [] pipe_write+0x274/0x580 Apr 14 02:03:37 opus kernel: [] do_sync_write+0x8d/0xd0 Apr 14 02:03:37 opus kernel: [] vfs_write+0xbd/0x1e0 Apr 14 02:03:37 opus kernel: [] SyS_write+0x58/0xb0 Apr 14 02:03:37 opus kernel: [] system_call_fastpath+0x16/0x1b Apr 14 02:03:37 opus kernel: INFO: task BackupPC_dump:5780 blocked for more than 120 seconds. Apr 14 02:03:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:03:37 opus kernel: BackupPC_dump D ffff88041e393680 0 5780 5722 0x00000080 Apr 14 02:03:37 opus kernel: ffff8803cb863bf0 0000000000000082 ffff8803cb863fd8 0000000000013680 Apr 14 02:03:37 opus kernel: ffff8803cb863fd8 0000000000013680 ffff8803ede16660 ffff8803f4aadb58 Apr 14 02:03:37 opus kernel: ffff8803f4aadb5c ffff8803ede16660 00000000ffffffff ffff8803f4aadb60 Apr 14 02:03:37 opus kernel: Call Trace: Apr 14 02:03:37 opus kernel: [] schedule_preempt_disabled+0x29/0x70 Apr 14 02:03:37 opus kernel: [] mutex_lock_slowpath+0xc5/0x1c0 Apr 14 02:03:37 opus kernel: [] mutex_lock+0x1f/0x2f Apr 14 02:03:37 opus kernel: [] zfs_mknode+0x13c/0xe50 [zfs] Apr 14 02:03:37 opus kernel: [] ? txg_rele_to_quiesce+0x33/0x40 [zfs] Apr 14 02:03:37 opus kernel: [] zfs_mkdir+0x479/0x5b0 [zfs] Apr 14 02:03:37 opus kernel: [] zpl_mkdir+0xaf/0x1d0 [zfs] Apr 14 02:03:37 opus kernel: [] vfs_mkdir+0xb7/0x160 Apr 14 02:03:37 opus kernel: [] SyS_mkdirat+0x6f/0xe0 Apr 14 02:03:37 opus kernel: [] SyS_mkdir+0x19/0x20 Apr 14 02:03:37 opus kernel: [] system_call_fastpath+0x16/0x1b Apr 14 02:04:32 opus smbd[5810]: [2015/04/14 02:04:32.985768, 0] ../source3/printing/print_cups.c:151(cups_connect) Apr 14 02:04:32 opus smbd[5810]: Unable to connect to CUPS server localhost:631 - Transport endpoint is not connected Apr 14 02:04:32 opus smbd[2304]: [2015/04/14 02:04:32.986841, 0] ../source3/printing/print_cups.c:528(cups_async_callback) Apr 14 02:04:32 opus smbd[2304]: failed to retrieve printer list: NT_STATUS_UNSUCCESSFUL Apr 14 02:05:37 opus kernel: INFO: task khugepaged:48 blocked for more than 120 seconds. Apr 14 02:05:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:05:37 opus kernel: khugepaged D ffff88041e213680 0 48 2 0x00000000 Apr 14 02:05:37 opus kernel: ffff880407a63870 0000000000000046 ffff880407a63fd8 0000000000013680 Apr 14 02:05:37 opus kernel: ffff880407a63fd8 0000000000013680 ffff880407d8db00 ffff8803f4aadb58 Apr 14 02:05:37 opus kernel: ffff8803f4aadb5c ffff880407d8db00 00000000ffffffff ffff8803f4aadb60 Apr 14 02:05:37 opus kernel: Call Trace: Apr 14 02:05:37 opus kernel: [] schedule_preempt_disabled+0x29/0x70 Apr 14 02:05:37 opus kernel: [] __mutex_lock_slowpath+0xc5/0x1c0 Apr 14 02:05:37 opus kernel: [] ? spl_kmem_free+0x32/0x50 [spl] Apr 14 02:05:37 opus kernel: [] mutex_lock+0x1f/0x2f Apr 14 02:05:37 opus kernel: [] zfs_zinactive+0x52/0x180 [zfs] Apr 14 02:05:37 opus kernel: [] zfs_inactive+0x64/0x230 [zfs] Apr 14 02:05:37 opus kernel: [] zpl_evict_inode+0x43/0x60 [zfs] Apr 14 02:05:37 opus kernel: [] evict+0xa7/0x170 Apr 14 02:05:37 opus kernel: [] dispose_list+0x3e/0x50 Apr 14 02:05:37 opus kernel: [] prune_icache_sb+0x163/0x320 Apr 14 02:05:37 opus kernel: [] prune_super+0xd6/0x1a0 Apr 14 02:05:37 opus kernel: [] shrink_slab+0x165/0x300 Apr 14 02:05:37 opus kernel: [] ? zone_watermark_ok+0x1f/0x30 Apr 14 02:05:37 opus kernel: [] do_try_to_free_pages+0x3c2/0x4e0 Apr 14 02:05:37 opus kernel: [] try_to_free_pages+0xfc/0x180 Apr 14 02:05:37 opus kernel: [] alloc_pages_nodemask+0x7fd/0xb90 Apr 14 02:05:37 opus kernel: [] khugepaged_scan_mm_slot+0x3c9/0xca0 Apr 14 02:05:37 opus kernel: [] khugepaged+0x25f/0x4a0 Apr 14 02:05:37 opus kernel: [] ? wake_up_bit+0x30/0x30 Apr 14 02:05:37 opus kernel: [] ? khugepaged_scan_mm_slot+0xca0/0xca0 Apr 14 02:05:37 opus kernel: [] kthread+0xcf/0xe0 Apr 14 02:05:37 opus kernel: [] ? kthread_create_on_node+0x140/0x140 Apr 14 02:05:37 opus kernel: [] ret_from_fork+0x7c/0xb0 Apr 14 02:05:37 opus kernel: [] ? kthread_create_on_node+0x140/0x140 Apr 14 02:05:37 opus kernel: INFO: task txg_quiesce:1305 blocked for more than 120 seconds. Apr 14 02:05:37 opus kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Apr 14 02:05:37 opus kernel: txg_quiesce D ffff88041e393680 0 1305 2 0x00000000 Apr 14 02:05:37 opus kernel: ffff8803f7f67d40 0000000000000046 ffff8803f7f67fd8 0000000000013680 Apr 14 02:05:37 opus kernel: ffff8803f7f67fd8 0000000000013680 ffff8803feb2c440 ffff8804071bd7f0 Apr 14 02:05:37 opus kernel: ffff8804071bd6e0 ffff8804071bd7f8 ffff8804071bd708 0000000000000000 Apr 14 02:05:37 opus kernel: Call Trace: Apr 14 02:05:37 opus kernel: [] schedule+0x29/0x70 Apr 14 02:05:37 opus kernel: [] cv_wait_common+0x125/0x150 [spl] Apr 14 02:05:37 opus kernel: [] ? wake_up_bit+0x30/0x30 Apr 14 02:05:37 opus kernel: [] cv_wait+0x15/0x20 [spl] Apr 14 02:05:37 opus kernel: [] txg_quiesce_thread+0x2bb/0x3a0 [zfs] Apr 14 02:05:37 opus kernel: [] ? txg_init+0x2b0/0x2b0 [zfs] Apr 14 02:05:37 opus kernel: [] thread_generic_wrapper+0x71/0x80 [spl] Apr 14 02:05:37 opus kernel: [] ? thread_exit+0x20/0x20 [spl] Apr 14 02:05:37 opus kernel: [] kthread+0xcf/0xe0 Apr 14 02:05:37 opus kernel: [] ? kthread_create_on_node+0x140/0x140 Apr 14 02:05:37 opus kernel: [] ret_from_fork+0x7c/0xb0 Apr 14 02:05:37 opus kernel: [] ? kthread_create_on_node+0x140/0x140

dweeezil commented 9 years ago

Based on the xattr calls, I have a feeling this will be addressed by #3293 once it's finished. Unfortunately, the original "post-kmem-rework deadlock" patch which was committed shortly before 0.6.4 missed a few potentially deadlock-causing paths.

rgmiller commented 9 years ago

Fair enough. Do you need me to post any more details (zdb output, system config, etc..)?

Also, is there a work-around? Or should I revert to 0.6.3?

dweeezil commented 9 years ago

If you can apply the current patch (1cf8f03) in #3293, it should fix your problem. There's a chance it might be expanded before it's committed but it does currently have the xattr parts covered which I think are causing your problem.

You might also be able to avoid the problem in the first place by capping the ARC size: set both zfs_arc_max and, likely more importantly, zfs_arc_meta_limit.

In either case, it would be interesting to see arcstats from your previous deadlock.

rgmiller commented 9 years ago

I'll turn BackupPC back on and let it deadlock again. Then I can post the arc stats. It'll probably take a few hours.

(Just so I'm clear: you're referring to the output of arcstat.py, yes? Any special arguments you want me to run it with?)

Thanks for the quick replies, BTW.

dweeezil commented 9 years ago

@rgmiller Please cat /proc/spl/kstat/zfs/arcstats and post the output. Might as well also do cat /proc/spl/kmem/slab and cat /proc/slabinfo while you're at it.

dweeezil commented 9 years ago

@rgmiller To be clear, it's more of a curiosity to get that information and better understand the memory situation on the machine which drove it into reclaim. The underlying deadlock is well-understood but, unfortunately, wasn't completely fixed prior to the 0.6.4 tag.

rgmiller commented 9 years ago

OK, it hung up again at 2AM this morning. Here's the stats data. (Note: I cat'd the files at about 8AM, after the server had been hung for about 6 hours. I don't know if that's important or not.) I also have stats taken from last night when the server was healthy if you want to compare. Let me know if you want me to post them.

arcstats: 5 1 0x01 86 4128 538148400 61406558270590 name type data hits 4 21149231 misses 4 9443130 demand_data_hits 4 8057545 demand_data_misses 4 2749947 demand_metadata_hits 4 11268250 demand_metadata_misses 4 3576303 prefetch_data_hits 4 39464 prefetch_data_misses 4 2538095 prefetch_metadata_hits 4 1783972 prefetch_metadata_misses 4 578785 mru_hits 4 8420449 mru_ghost_hits 4 1363213 mfu_hits 4 10905979 mfu_ghost_hits 4 1658027 deleted 4 4782484 recycle_miss 4 1079903 mutex_miss 4 124 evict_skip 4 170841187 evict_l2_cached 4 50863008768 evict_l2_eligible 4 301446057984 evict_l2_ineligible 4 47985894400 hash_elements 4 2396354 hash_elements_max 4 2574784 hash_collisions 4 4798003 hash_chains 4 664041 hash_chain_max 4 10 p 4 3590213120 c 4 4882926920 c_min 4 4194304 c_max 4 8152113152 size 4 5998009000 hdr_size 4 584376048 data_size 4 0 meta_size 4 2228724736 other_size 4 2934114336 anon_size 4 32768 anon_evict_data 4 0 anon_evict_metadata 4 0 mru_size 4 1945377792 mru_evict_data 4 0 mru_evict_metadata 4 9485312 mru_ghost_size 4 2937464320 mru_ghost_evict_data 4 0 mru_ghost_evict_metadata 4 2937464320 mfu_size 4 283314176 mfu_evict_data 4 0 mfu_evict_metadata 4 0 mfu_ghost_size 4 1940601856 mfu_ghost_evict_data 4 956668416 mfu_ghost_evict_metadata 4 983933440 l2_hits 4 1929810 l2_misses 4 7513265 l2_feeds 4 63407 l2_rw_clash 4 6 l2_read_bytes 4 3941563904 l2_write_bytes 4 25021215232 l2_writes_sent 4 7037 l2_writes_done 4 7037 l2_writes_error 4 0 l2_writes_hdr_miss 4 8 l2_evict_lock_retry 4 0 l2_evict_reading 4 0 l2_free_on_write 4 31332 l2_cdata_free_on_write 4 83 l2_abort_lowmem 4 222 l2_cksum_bad 4 0 l2_io_error 4 0 l2_size 4 33688148480 l2_asize 4 24259862528 l2_hdr_size 4 250793880 l2_compress_successes 4 565507 l2_compress_zeros 4 18025 l2_compress_failures 4 170195 memory_throttle_count 4 0 duplicate_buffers 4 0 duplicate_buffers_size 4 0 duplicate_reads 4 0 memory_direct_count 4 3272 memory_indirect_count 4 98658 arc_no_grow 4 0 arc_tempreserve 4 0 arc_loaned_bytes 4 0 arc_prune 4 1791 arc_meta_used 4 5998009000 arc_meta_limit 4 6114084864 arc_meta_max 4 6121255064

slabinfo: slabinfo - version: 2.1

name : tunables : slabdata

nf_conntrack_ffffffff819e07c0 375 375 320 25 2 : tunables 0 0 0 : slabdata 15 15 0 kvm_async_pf 0 0 136 30 1 : tunables 0 0 0 : slabdata 0 0 0 kvm_vcpu 0 0 16256 2 8 : tunables 0 0 0 : slabdata 0 0 0 kvm_mmu_page_header 0 0 168 24 1 : tunables 0 0 0 : slabdata 0 0 0 nfsd4_delegations 17 17 240 17 1 : tunables 0 0 0 : slabdata 1 1 0 nfsd4_openowners 36 36 440 18 2 : tunables 0 0 0 : slabdata 2 2 0 rpc_inode_cache 75 75 640 25 4 : tunables 0 0 0 : slabdata 3 3 0 xfs_icr 0 0 144 28 1 : tunables 0 0 0 : slabdata 0 0 0 xfs_ili 12952 15600 152 26 1 : tunables 0 0 0 : slabdata 600 600 0 xfs_inode 28272 28272 1024 16 4 : tunables 0 0 0 : slabdata 1767 1767 0 xfs_efd_item 140 140 400 20 2 : tunables 0 0 0 : slabdata 7 7 0 xfs_da_state 64 64 488 16 2 : tunables 0 0 0 : slabdata 4 4 0 xfs_btree_cur 76 76 208 19 1 : tunables 0 0 0 : slabdata 4 4 0 xfs_log_ticket 88 88 184 22 1 : tunables 0 0 0 : slabdata 4 4 0 kcopyd_job 0 0 3312 9 8 : tunables 0 0 0 : slabdata 0 0 0 dm_uevent 0 0 2608 12 8 : tunables 0 0 0 : slabdata 0 0 0 dm_rq_target_io 0 0 424 19 2 : tunables 0 0 0 : slabdata 0 0 0 zil_lwb_cache 1 20 200 20 1 : tunables 0 0 0 : slabdata 1 1 0 dmu_buf_impl_t 2068808 2545790 304 26 2 : tunables 0 0 0 : slabdata 97915 97915 0 dnode_t 1952083 2035584 864 18 4 : tunables 0 0 0 : slabdata 113088 113088 0 zio_buf_16384 136035 138266 16384 2 8 : tunables 0 0 0 : slabdata 69134 69134 0 zio_buf_14336 2 2 14336 2 8 : tunables 0 0 0 : slabdata 1 1 0 zio_buf_12288 2 2 12288 2 8 : tunables 0 0 0 : slabdata 1 1 0 zio_buf_10240 6 6 10240 3 8 : tunables 0 0 0 : slabdata 2 2 0 zio_buf_7168 8 8 7168 4 8 : tunables 0 0 0 : slabdata 2 2 0 zio_buf_6144 0 0 6144 5 8 : tunables 0 0 0 : slabdata 0 0 0 zio_buf_5120 6 6 5120 6 8 : tunables 0 0 0 : slabdata 1 1 0 zio_buf_3584 0 0 3584 9 8 : tunables 0 0 0 : slabdata 0 0 0 zio_buf_3072 10 10 3072 10 8 : tunables 0 0 0 : slabdata 1 1 0 zio_buf_2560 24 24 2560 12 8 : tunables 0 0 0 : slabdata 2 2 0 zio_buf_1536 42 42 1536 21 8 : tunables 0 0 0 : slabdata 2 2 0 zio_cache 145 145 1096 29 8 : tunables 0 0 0 : slabdata 5 5 0 ddt_entry_cache 108 108 448 18 2 : tunables 0 0 0 : slabdata 6 6 0 UDPLITEv6 0 0 1152 28 8 : tunables 0 0 0 : slabdata 0 0 0 UDPv6 112 112 1152 28 8 : tunables 0 0 0 : slabdata 4 4 0 tw_sock_TCPv6 32 32 256 16 1 : tunables 0 0 0 : slabdata 2 2 0 TCPv6 64 64 2048 16 8 : tunables 0 0 0 : slabdata 4 4 0 cfq_queue 357 357 232 17 1 : tunables 0 0 0 : slabdata 21 21 0 bsg_cmd 2396384 2573454 312 26 2 : tunables 0 0 0 : slabdata 98979 98979 0 mqueue_inode_cache 18 18 896 18 4 : tunables 0 0 0 : slabdata 1 1 0 hugetlbfs_inode_cache 52 52 608 26 4 : tunables 0 0 0 : slabdata 2 2 0 configfs_dir_cache 0 0 88 46 1 : tunables 0 0 0 : slabdata 0 0 0 dquot 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 kioctx 28 28 576 28 4 : tunables 0 0 0 : slabdata 1 1 0 pid_namespace 0 0 2176 15 8 : tunables 0 0 0 : slabdata 0 0 0 posix_timers_cache 0 0 248 16 1 : tunables 0 0 0 : slabdata 0 0 0 UDP-Lite 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 ip_fib_trie 146 146 56 73 1 : tunables 0 0 0 : slabdata 2 2 0 RAW 255 255 960 17 4 : tunables 0 0 0 : slabdata 15 15 0 UDP 64 64 1024 16 4 : tunables 0 0 0 : slabdata 4 4 0 tw_sock_TCP 96 96 256 16 1 : tunables 0 0 0 : slabdata 6 6 0 TCP 68 68 1920 17 8 : tunables 0 0 0 : slabdata 4 4 0 blkdev_queue 48 48 2016 16 8 : tunables 0 0 0 : slabdata 3 3 0 blkdev_requests 3163 3192 384 21 2 : tunables 0 0 0 : slabdata 152 152 0 blkdev_ioc 575508 1396551 104 39 1 : tunables 0 0 0 : slabdata 35809 35809 0 fsnotify_event_holder 121210 121210 24 170 1 : tunables 0 0 0 : slabdata 713 713 0 fsnotify_event 578 578 120 34 1 : tunables 0 0 0 : slabdata 17 17 0 sock_inode_cache 400 400 640 25 4 : tunables 0 0 0 : slabdata 16 16 0 net_namespace 0 0 4224 7 8 : tunables 0 0 0 : slabdata 0 0 0 shmem_inode_cache 1728 1728 680 24 4 : tunables 0 0 0 : slabdata 72 72 0 Acpi-ParseExt 4200 4200 72 56 1 : tunables 0 0 0 : slabdata 75 75 0 Acpi-Namespace 1863655 1863846 40 102 1 : tunables 0 0 0 : slabdata 18273 18273 0 taskstats 96 96 328 24 2 : tunables 0 0 0 : slabdata 4 4 0 proc_inode_cache 2870 2976 656 24 4 : tunables 0 0 0 : slabdata 124 124 0 sigqueue 150 150 160 25 1 : tunables 0 0 0 : slabdata 6 6 0 bdev_cache 76 76 832 19 4 : tunables 0 0 0 : slabdata 4 4 0 sysfs_dir_cache 28349 28656 112 36 1 : tunables 0 0 0 : slabdata 796 796 0 inode_cache 12330 13527 592 27 4 : tunables 0 0 0 : slabdata 501 501 0 dentry 111709 125853 192 21 1 : tunables 0 0 0 : slabdata 5993 5993 0 iint_cache 0 0 80 51 1 : tunables 0 0 0 : slabdata 0 0 0 selinux_inode_security 620859 1428663 80 51 1 : tunables 0 0 0 : slabdata 28013 28013 0 buffer_head 69266 73047 104 39 1 : tunables 0 0 0 : slabdata 1873 1873 0 vm_area_struct 15483 15534 216 18 1 : tunables 0 0 0 : slabdata 863 863 0 mm_struct 160 160 1600 20 8 : tunables 0 0 0 : slabdata 8 8 0 files_cache 125 125 640 25 4 : tunables 0 0 0 : slabdata 5 5 0 signal_cache 629 728 1152 28 8 : tunables 0 0 0 : slabdata 26 26 0 sighand_cache 535 540 2112 15 8 : tunables 0 0 0 : slabdata 36 36 0 task_xstate 874 874 832 19 4 : tunables 0 0 0 : slabdata 46 46 0 task_struct 537 638 2912 11 8 : tunables 0 0 0 : slabdata 58 58 0 anon_vma 7658 8192 64 64 1 : tunables 0 0 0 : slabdata 128 128 0 shared_policy_node 2563 2890 48 85 1 : tunables 0 0 0 : slabdata 34 34 0 numa_policy 31 31 264 31 2 : tunables 0 0 0 : slabdata 1 1 0 radix_tree_node 27188 27188 584 28 4 : tunables 0 0 0 : slabdata 971 971 0 idr_layer_cache 270 270 2112 15 8 : tunables 0 0 0 : slabdata 18 18 0 dma-kmalloc-8192 0 0 8192 4 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-4096 0 0 4096 8 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-2048 0 0 2048 16 8 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-1024 0 0 1024 16 4 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-512 0 0 512 16 2 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-256 0 0 256 16 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-128 0 0 128 32 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-64 0 0 64 64 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-32 0 0 32 128 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-16 0 0 16 256 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-8 0 0 8 512 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-192 0 0 192 21 1 : tunables 0 0 0 : slabdata 0 0 0 dma-kmalloc-96 0 0 96 42 1 : tunables 0 0 0 : slabdata 0 0 0 kmalloc-8192 132489 135348 8192 4 8 : tunables 0 0 0 : slabdata 33837 33837 0 kmalloc-4096 358 536 4096 8 8 : tunables 0 0 0 : slabdata 67 67 0 kmalloc-2048 593 688 2048 16 8 : tunables 0 0 0 : slabdata 43 43 0 kmalloc-1024 15216 15872 1024 16 4 : tunables 0 0 0 : slabdata 992 992 0 kmalloc-512 1934671 2431984 512 16 2 : tunables 0 0 0 : slabdata 151999 151999 0 kmalloc-256 4940 8032 256 16 1 : tunables 0 0 0 : slabdata 502 502 0 kmalloc-192 3675 4620 192 21 1 : tunables 0 0 0 : slabdata 220 220 0 kmalloc-128 803971 1121536 128 32 1 : tunables 0 0 0 : slabdata 35048 35048 0 kmalloc-96 527472 1518006 96 42 1 : tunables 0 0 0 : slabdata 36143 36143 0 kmalloc-64 1173715 3267264 64 64 1 : tunables 0 0 0 : slabdata 51051 51051 0 kmalloc-32 2747321 3440256 32 128 1 : tunables 0 0 0 : slabdata 26877 26877 0 kmalloc-16 54784 54784 16 256 1 : tunables 0 0 0 : slabdata 214 214 0 kmalloc-8 1077442 1747968 8 512 1 : tunables 0 0 0 : slabdata 3414 3414 0 kmem_cache_node 320 320 64 64 1 : tunables 0 0 0 : slabdata 5 5 0 kmem_cache 144 144 256 16 1 : tunables 0 0 0 : slabdata 9 9 0

slab: --------------------- cache ------------------------------------------------------- ----- slab ------ ---- object ----- --- emergency --- name flags size alloc slabsize objsize total alloc max total alloc max dlock alloc max spl_vn_cache 0x00020 8192 112 8192 112 1 1 2 42 1 84 0 0 0 spl_vn_file_cache 0x00020 8192 120 8192 120 1 1 4 42 1 168 0 0 0 spl_zlib_workspace_cache 0x00240 0 0 2145216 268104 0 0 0 0 0 0 0 0 0 ddt_cache 0x00040 797184 497280 199296 24864 4 4 6 32 20 48 0 0 0 zio_buf_20480 0x00042 602112 389120 200704 20480 3 3 54 24 19 432 0 0 0 zio_data_buf_20480 0x00042 0 0 200704 20480 0 0 577 0 0 4616 0 0 0 zio_buf_24576 0x00042 466944 245760 233472 24576 2 2 29 16 10 232 1 0 1 zio_data_buf_24576 0x00042 0 0 233472 24576 0 0 730 0 0 5840 0 0 0 zio_buf_28672 0x00042 532480 286720 266240 28672 2 2 97 16 10 776 0 0 0 zio_data_buf_28672 0x00042 0 0 266240 28672 0 0 363 0 0 2904 0 0 0 zio_buf_32768 0x00042 897024 557056 299008 32768 3 3 736 24 17 5888 0 0 0 zio_data_buf_32768 0x00042 0 0 299008 32768 0 0 159 0 0 1272 0 0 0 zio_buf_36864 0x00042 995328 737280 331776 36864 3 3 77 24 20 616 0 0 0 zio_data_buf_36864 0x00042 0 0 331776 36864 0 0 121 0 0 968 0 0 0 zio_buf_40960 0x00042 729088 655360 364544 40960 2 2 51 16 16 408 0 0 0 zio_data_buf_40960 0x00042 0 0 364544 40960 0 0 101 0 0 808 0 0 0 zio_buf_45056 0x00042 794624 675840 397312 45056 2 2 40 16 15 320 0 0 0 zio_data_buf_45056 0x00042 0 0 397312 45056 0 0 77 0 0 616 0 0 0 zio_buf_49152 0x00042 430080 393216 430080 49152 1 1 34 8 8 272 0 0 0 zio_data_buf_49152 0x00042 0 0 430080 49152 0 0 123 0 0 984 0 0 0 zio_buf_53248 0x00042 462848 53248 462848 53248 1 1 32 8 1 256 0 0 0 zio_data_buf_53248 0x00042 0 0 462848 53248 0 0 56 0 0 448 0 0 0 zio_buf_57344 0x00042 991232 917504 495616 57344 2 2 33 16 16 264 0 0 0 zio_data_buf_57344 0x00042 0 0 495616 57344 0 0 69 0 0 552 0 0 0 zio_buf_61440 0x00042 1056768 552960 528384 61440 2 2 66 16 9 528 0 0 0 zio_data_buf_61440 0x00042 0 0 528384 61440 0 0 49 0 0 392 0 0 0 zio_buf_65536 0x00042 561152 65536 561152 65536 1 1 45 8 1 360 0 0 0 zio_data_buf_65536 0x00042 0 0 561152 65536 0 0 56 0 0 448 0 0 0 zio_buf_69632 0x00042 593920 348160 593920 69632 1 1 36 8 5 288 0 0 0 zio_data_buf_69632 0x00042 0 0 593920 69632 0 0 68 0 0 544 0 0 0 zio_buf_73728 0x00042 626688 516096 626688 73728 1 1 59 8 7 472 0 0 0 zio_data_buf_73728 0x00042 0 0 626688 73728 0 0 63 0 0 504 0 0 0 zio_buf_77824 0x00042 0 0 659456 77824 0 0 88 0 0 704 0 0 0 zio_data_buf_77824 0x00042 0 0 659456 77824 0 0 73 0 0 584 0 0 0 zio_buf_81920 0x00042 0 0 692224 81920 0 0 87 0 0 696 0 0 0 zio_data_buf_81920 0x00042 0 0 692224 81920 0 0 71 0 0 568 0 0 0 zio_buf_86016 0x00042 724992 688128 724992 86016 1 1 32 8 8 256 0 0 0 zio_data_buf_86016 0x00042 0 0 724992 86016 0 0 62 0 0 496 0 0 0 zio_buf_90112 0x00042 0 0 757760 90112 0 0 41 0 0 328 0 0 0 zio_data_buf_90112 0x00042 0 0 757760 90112 0 0 55 0 0 440 0 0 0 zio_buf_94208 0x00042 790528 565248 790528 94208 1 1 21 8 6 168 0 0 0 zio_data_buf_94208 0x00042 0 0 790528 94208 0 0 34 0 0 272 0 0 0 zio_buf_98304 0x00042 823296 688128 823296 98304 1 1 25 8 7 200 0 0 0 zio_data_buf_98304 0x00042 0 0 823296 98304 0 0 38 0 0 304 0 0 0 zio_buf_102400 0x00042 856064 204800 856064 102400 1 1 17 8 2 136 0 0 0 zio_data_buf_102400 0x00042 0 0 856064 102400 0 0 28 0 0 224 0 0 0 zio_buf_106496 0x00042 0 0 888832 106496 0 0 24 0 0 192 0 0 0 zio_data_buf_106496 0x00042 0 0 888832 106496 0 0 41 0 0 328 0 0 0 zio_buf_110592 0x00042 921600 663552 921600 110592 1 1 19 8 6 152 0 0 0 zio_data_buf_110592 0x00042 0 0 921600 110592 0 0 25 0 0 200 0 0 0 zio_buf_114688 0x00042 954368 114688 954368 114688 1 1 30 8 1 240 0 0 0 zio_data_buf_114688 0x00042 0 0 954368 114688 0 0 22 0 0 176 0 0 0 zio_buf_118784 0x00042 987136 950272 987136 118784 1 1 12 8 8 96 0 0 0 zio_data_buf_118784 0x00042 0 0 987136 118784 0 0 19 0 0 152 0 0 0 zio_buf_122880 0x00042 0 0 1019904 122880 0 0 14 0 0 112 0 0 0 zio_data_buf_122880 0x00042 0 0 1019904 122880 0 0 20 0 0 160 0 0 0 zio_buf_126976 0x00042 2105344 1269760 1052672 126976 2 2 17 16 10 136 1 0 1 zio_data_buf_126976 0x00042 0 0 1052672 126976 0 0 15 0 0 120 0 0 0 zio_buf_131072 0x00042 2170880 917504 1085440 131072 2 2 268 16 7 2144 0 0 0 zio_data_buf_131072 0x00042 4341760 1703936 1085440 131072 4 4 6641 32 13 53128 0 0 0 zfs_znode_cache 0x00020 1288568832 630604424 8192 1096 157296 157296 275578 1101072 575369 1929046 0 0 0

rgmiller commented 9 years ago

I rebuilt the zfs-dkms RPM using the latest code on the zfs-0.6.4-release branch, and it's been running stably since Friday, so I think we can mark this issue closed.

behlendorf commented 9 years ago

@rgmiller Thanks for following up and I'm glad to hear it's working well for you now. I've just tagged 0.6.4.1 with these fixes so you can move back to the stable repo.