brendonhumphrey
commented
10 years ago Hi,
Thanks a lot for going to the trouble to report this to us. Would you be able to post the entire spindump output here please? We need to examine the kernel threads to determine if this is a ifs deadlock.
Thanks Brendon
On 12 Mar 2014, at 3:08 pm, dmzimmerman notifications@github.com wrote:
On a build from this morning of the latest code pulled by the "zfsadm" script, I created a single disk pool on a FireWire 800-connected disk (the goal being to migrate data from my HFS+ RAID0+1 before swapping the ZFS disk into the Mac Pro, then mirroring it to another ZFS disk) with
zpool create -f -O compression=lz4 -O casesensitivity=insensitive -O normalization=formD -O atime=off -o ashift=12 zfs disk7 (yes, I called my pool "zfs")
I then created some filesystems on the pool, explicitly disabled Spotlight on all of them (including /zfs), and attempted to use rsync 3 (from MacPorts) to copy about 3 TB of data to the pool, as follows:
sudo time /opt/local/bin/rsync --stats -aNHAX --fileflags --safe-links --partial --progress --delete --exclude-from=rsync-mirror.excludes --rsync-path=/opt/local/bin/rsync /Volumes/Zocalo/ /zfs After about 500GB had been copied, all ZFS operations (including the file copy and "ls" on the filesystem) exhibited hangs. I got the following samples from spindump for a hung "zpool" and a hung "ls":
Date/Time: 2014-03-11 19:42:40 -0700 OS Version: 10.9.2 (Build 13C64) Architecture: x86_64 Report Version: 18
Command: zpool Path: /usr/local/sbin/zpool Version: ??? (???) Parent: bash [3448]
PID: 3462 Event: manual Duration: 10.01s Steps: 1001 (10ms sampling interval)
Hardware model: MacPro5,1 Active cpus: 24 Fan speed: 799 rpm
Free pages: 16941 pages (-2978) Pageins: 75 pages Pageouts: 0 pages Swapins: 0 pages Swapouts: 0 pages
Process: zpool [3462] Path: /usr/local/sbin/zpool Architecture: x86_64 Parent: bash [3448] UID: 0 Task size: 267 pages Importance: Adaptive, Donating
Thread 0x2d042 DispatchQueue 1 priority 31
1001 start + 1 (libdyld.dylib) [0x7fff8feaa5fd] 1001 main + 400 (zpool) [0x100b85be0] 1001 zpool_do_status + 484 (zpool) [0x100b88284] 1001 for_each_pool + 60 (zpool) [0x100b8147c] 1001 pool_list_get + 228 (zpool) [0x100b80f84] 1001 zpool_iter + 173 (libzfs.1.dylib) [0x100d9928d] 1001 zpool_open_silent + 140 (libzfs.1.dylib) [0x100db432c] 1001 zpool_refresh_stats + 279 (libzfs.1.dylib) [0x100d98e57] 1001 zfs_ioctl + 62 (libzfs.1.dylib) [0x100dcd1de] 1001 __ioctl + 10 (libsystem_kernel.dylib) [0x7fff8c0a624a] 1001 hndl_unix_scall64 + 22 (mach_kernel) [0xffffff80002f3b26] 1001 unix_syscall64 + 499 (mach_kernel) [0xffffff800063e363] 1001 ioctl + 1126 (mach_kernel) [0xffffff80005f09e6] 1001 fo_ioctl + 67 (mach_kernel) [0xffffff80005bf0a3] 1001 ??? (mach_kernel + 2046689) [0xffffff80003f3ae1] 1001 VNOP_IOCTL + 336 (mach_kernel) [0xffffff80003fdc90] 1001 spec_ioctl + 287 (mach_kernel) [0xffffff800040cfdf] 1001 zfsdev_ioctl + 1240 (zfs_ioctl.c:6151 in zfs) [0xffffff7f80d96af8] 1001 zfs_ioc_pool_stats + 55 (zfs_ioctl.c:1507 in zfs) [0xffffff7f80d97577] 1001 spa_get_stats + 463 (spa.c:3205 in zfs) [0xffffff7f80d4a8bf] 1001 spa_add_feature_stats + 150 (spa.c:3140 in zfs) [0xffffff7f80d4aee6] 1001 zap_cursor_retrieve + 128 (zap_micro.c:1260 in zfs) [0xffffff7f80d809d0] 1001 zap_lockdir + 95 (zap_micro.c:456 in zfs) [0xffffff7f80d7e43f] 1001 dmu_buf_hold + 234 (dmu.c:146 in zfs) [0xffffff7f80cf785a] 1001 dbuf_read + 713 (dbuf.c:699 in zfs) [0xffffff7f80ced6a9] 1001 zio_wait + 115 (zio.c:1428 in zfs) [0xffffff7f80dd2aa3] 1001 spl_cv_wait + 51 (spl-condvar.c:67 in spl) [0xffffff7f80cd1853] 1001 msleep + 114 (mach_kernel) [0xffffff80005e1802] 1001 ??? (mach_kernel + 4068821) [0xffffff80005e15d5] 1001 lck_mtx_sleep + 78 (mach_kernel) [0xffffff800022d1be] 1001 thread_block_reason + 204 (mach_kernel) [0xffffff8000235d8c] 1001 ??? (mach_kernel + 223627) [0xffffff800023698b] *1001 machine_switch_context + 354 (mach_kernel) [0xffffff80002d7e42]Binary Images: 0x100b80000 - 0x100b99ff7 zpool (0) <5E63A16D-36D8-31B9-8F4D-264342F92E3C> /usr/local/sbin/zpool 0x100d96000 - 0x100ddafff libzfs.1.dylib (0) <7D6D0BDF-06EB-32E1-B234-AA9981C51381> /usr/local/lib/libzfs.1.dylib 0x7fff8c091000 - 0x7fff8c0adff7 libsystem_kernel.dylib (2422.90.20) <20E00C54-9222-359F-BD98-CB79ABED769A> /usr/lib/system/libsystem_kernel.dylib 0x7fff8fea7000 - 0x7fff8feaaff7 libdyld.dylib (239.4)
/usr/lib/system/libdyld.dylib 0xffffff7f80cd1000 - 0xffffff7f80cd5ff5 net.lundman.spl 1.0 (1.0.0) /System/Library/Extensions/spl.kext/Contents/MacOS/spl 0xffffff7f80cde000 - 0xffffff7f80e5bfff net.lundman.zfs 1.0 (1.0.0) <048ADB8C-44FA-39AC-9EE6-CCF847F59E74> /System/Library/Extensions/zfs.kext/Contents/MacOS/zfs *0xffffff8000200000 - 0xffffff80007a513a mach_kernel (2422.90.20) <9FEA8EDC-B629-3ED2-A1A3-6521A1885953> /mach_kernel Date/Time: 2014-03-11 19:42:59 -0700 OS Version: 10.9.2 (Build 13C64) Architecture: x86_64 Report Version: 18Command: ls Path: /bin/ls Version: ??? (???) Parent: sudo [3115]
PID: 3116 Event: manual Duration: 10.01s Steps: 994 (10ms sampling interval)
Hardware model: MacPro5,1 Active cpus: 24 Fan speed: 799 rpm
Free pages: 26040 pages (-4422) Pageins: 28 pages Pageouts: 0 pages Swapins: 0 pages Swapouts: 0 pages
Process: ls [3116] Path: /bin/ls Architecture: x86_64 Parent: sudo [3115] UID: 0 Task size: 150 pages Importance: Adaptive, Donating
Thread 0x2886d DispatchQueue 1 priority 31
994 start + 1 (libdyld.dylib) [0x7fff8feaa5fd] 994 ??? (ls + 7340) [0x1099eacac] 994 ??? (ls + 8107) [0x1099eafab] 994 fts_children$INODE64 + 173 (libsystem_c.dylib) [0x7fff8f6607b0] 994 fts_build + 379 (libsystem_c.dylib) [0x7fff8f6601dd] 994 readdir$INODE64 + 35 (libsystem_c.dylib) [0x7fff8f688619] 994 __getdirentries64 + 10 (libsystem_kernel.dylib) [0x7fff8c0a60c2] 994 hndl_unix_scall64 + 22 (mach_kernel) [0xffffff80002f3b26] 994 unix_syscall64 + 499 (mach_kernel) [0xffffff800063e363] 994 getdirentries64 + 58 (mach_kernel) [0xffffff80003ecdfa] 994 ??? (mach_kernel + 2017743) [0xffffff80003ec9cf] 994 VNOP_READDIR + 134 (mach_kernel) [0xffffff80003ff586] 994 zfs_vnop_readdir + 133 (zfs_vnops_osx.c:533 in zfs) [0xffffff7f80dbb6b5] 994 zfs_readdir + 1177 (zfs_vnops.c:2639 in zfs) [0xffffff7f80db26b9] 994 zap_cursor_retrieve + 128 (zap_micro.c:1260 in zfs) [0xffffff7f80d809d0] 994 zap_lockdir + 95 (zap_micro.c:456 in zfs) [0xffffff7f80d7e43f] 994 dmu_buf_hold + 234 (dmu.c:146 in zfs) [0xffffff7f80cf785a] 994 dbuf_read + 713 (dbuf.c:699 in zfs) [0xffffff7f80ced6a9] 994 zio_wait + 115 (zio.c:1428 in zfs) [0xffffff7f80dd2aa3] 994 spl_cv_wait + 51 (spl-condvar.c:67 in spl) [0xffffff7f80cd1853] 994 msleep + 114 (mach_kernel) [0xffffff80005e1802] 994 ??? (mach_kernel + 4068821) [0xffffff80005e15d5] 994 lck_mtx_sleep + 78 (mach_kernel) [0xffffff800022d1be] 994 thread_block_reason + 204 (mach_kernel) [0xffffff8000235d8c] 994 ??? (mach_kernel + 223627) [0xffffff800023698b] *994 machine_switch_context + 354 (mach_kernel) [0xffffff80002d7e42]Binary Images: 0x1099e9000 - 0x1099edff5 ls (230)
/bin/ls 0x7fff8c091000 - 0x7fff8c0adff7 libsystem_kernel.dylib (2422.90.20) <20E00C54-9222-359F-BD98-CB79ABED769A> /usr/lib/system/libsystem_kernel.dylib 0x7fff8f65b000 - 0x7fff8f6e4ff7 libsystem_c.dylib (997.90.3) <6FD3A400-4BB2-3B95-B90C-BE6E9D0D78FA> /usr/lib/system/libsystem_c.dylib 0x7fff8fea7000 - 0x7fff8feaaff7 libdyld.dylib (239.4) /usr/lib/system/libdyld.dylib 0xffffff7f80cd1000 - 0xffffff7f80cd5ff5 net.lundman.spl 1.0 (1.0.0) /System/Library/Extensions/spl.kext/Contents/MacOS/spl 0xffffff7f80cde000 - 0xffffff7f80e5bfff net.lundman.zfs 1.0 (1.0.0) <048ADB8C-44FA-39AC-9EE6-CCF847F59E74> /System/Library/Extensions/zfs.kext/Contents/MacOS/zfs *0xffffff8000200000 - 0xffffff80007a513a mach_kernel (2422.90.20) <9FEA8EDC-B629-3ED2-A1A3-6521A1885953> /mach_kernel (Ive cut out the information about the other processes captured by spindump)I had similar issues the last time I tried to do this sort of copy to a pool on an internal disk, so I doubt it's the FireWire 800 at fault. Hopefully this is enough info to try to track it down. If you need more info, let me know.
Reply to this email directly or view it on GitHub.
dmzimmerman
lundman
ilovezfs
zigmoo
JMoVS
On a build from this morning of the latest code pulled by the "zfsadm" script, I created a single disk pool on a FireWire 800-connected disk (the goal being to migrate data from my HFS+ RAID0+1 before swapping the ZFS disk into the Mac Pro, then mirroring it to another ZFS disk) with
(yes, I called my pool "zfs")
I then created some filesystems on the pool, explicitly disabled Spotlight on all of them (including /zfs), and attempted to use rsync 3 (from MacPorts) to copy about 3 TB of data to the pool, as follows:
After about 500GB had been copied, all ZFS operations (including the file copy and "ls" on the filesystem) exhibited hangs. I got the following samples from spindump for a hung "zpool" and a hung "ls":
(I’ve cut out the information about the other processes captured by spindump)
I had similar issues the last time I tried to do this sort of copy to a pool on an internal disk, so I doubt it's the FireWire 800 at fault. Hopefully this is enough info to try to track it down. If you need more info, let me know.