search

openzfsonosx / zfs

OpenZFS on OS X
https://openzfsonosx.org/
Other
823 stars 72 forks source link

zfs unresponsive upon removing cache #736

Open whizzrd opened 4 years ago

whizzrd commented 4 years ago 
zfs -V
zfs-1.9.0-1
zfs-kmod-1.9.0-1
brew cask info openzfs
openzfs: 1.9.0,18
https://openzfsonosx.org/
/usr/local/Caskroom/openzfs/1.9.0,18 (19 files, 60.1MB)
From: https://github.com/Homebrew/homebrew-cask/blob/master/Casks/openzfs.rb
==> Name
OpenZFS on OS X
==> Artifacts
OpenZFS on OS X 1.9.0 High Sierra.pkg (Pkg)
disk images without partitioning schema via device names

ZFS_POOL_NAME=ZFS_POOL_IMG
hdiutil create -size 5g -volname ${ZFS_POOL_NAME}_00 -type SPARSE -layout NONE  ${ZFS_POOL_NAME}_00
hdiutil create -size 5g -volname ${ZFS_POOL_NAME}_01 -type SPARSE -layout NONE  ${ZFS_POOL_NAME}_01
hdiutil create -size 5g -volname ${ZFS_POOL_NAME}_02 -type SPARSE -layout NONE  ${ZFS_POOL_NAME}_02
hdiutil create -size 5g -volname ${ZFS_POOL_NAME}_03 -type SPARSE -layout NONE  ${ZFS_POOL_NAME}_03
attach images and create pool
ZFS_POOL_DEV_00=`hdiutil attach -nomount  ${ZFS_POOL_NAME}_00.sparseimage`
ZFS_POOL_DEV_01=`hdiutil attach -nomount  ${ZFS_POOL_NAME}_01.sparseimage`
ZFS_POOL_DEV_02=`hdiutil attach -nomount  ${ZFS_POOL_NAME}_02.sparseimage`
ZFS_POOL_DEV_03=`hdiutil attach -nomount  ${ZFS_POOL_NAME}_03.sparseimage`

sudo zpool create  -f -m /Volumes/${ZFS_POOL_NAME} ${ZFS_POOL_NAME} raidz ${ZFS_POOL_DEV_00} ${ZFS_POOL_DEV_01} ${ZFS_POOL_DEV_02} ${ZFS_POOL_DEV_03}

try to prevent spotlight from doing it's thing

sudo touch /Volumes/${ZFS_POOL_NAME}/.metadata_never_index

ram drive cache

ZFS_POOL_DEV_CACHE=`hdiutil attach -nomount ram://8388608`
zpool add -f ${ZFS_POOL_NAME} cache ${ZFS_POOL_DEV_CACHE}

unplug

sudo zpool export $ZFS_POOL_NAME
hdiutil detach ${ZFS_POOL_DEV_00}
hdiutil detach ${ZFS_POOL_DEV_01}
hdiutil detach ${ZFS_POOL_DEV_02}
hdiutil detach ${ZFS_POOL_DEV_03}
hdiutil detach ${ZFS_POOL_DEV_CACHE}

remount

ZFS_POOL_DEV_00=`hdiutil attach -nomount  ${ZFS_POOL_NAME}_00.sparseimage`
ZFS_POOL_DEV_01=`hdiutil attach -nomount  ${ZFS_POOL_NAME}_01.sparseimage`
ZFS_POOL_DEV_02=`hdiutil attach -nomount  ${ZFS_POOL_NAME}_02.sparseimage`
ZFS_POOL_DEV_03=`hdiutil attach -nomount  ${ZFS_POOL_NAME}_03.sparseimage`
ZFS_POOL_DEV_CACHE=`hdiutil attach -nomount ram://8388608`
sudo zpool import -d /dev ${ZFS_POOL_NAME}

remove cache


zpool remove $ZFS_POOL_NAME ${ZFS_POOL_DEV_CACHE}
zfs now unresponsive (never returns from remove command)
log stream --source --predicate 'senderImagePath CONTAINS "zfs"'
Filtering the log data using "senderImagePath CONTAINS "zfs""
Timestamp                       Thread     Type        Activity             PID    TTL  
2019-10-23 08:37:06.375500+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:38:23.827436+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 7610368
2019-10-23 08:38:29.325241+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 9707520
2019-10-23 08:38:32.128586+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 7602176
2019-10-23 08:39:47.670499+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 16293888
2019-10-23 08:40:00.558575+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 14327808
2019-10-23 08:40:03.303979+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> SPL: arc_reclaim_thread: post-reap <private> post-evict 0 adjusted 0 pre-adjust 4378624 to-free 122404864 pressure 131072
2019-10-23 08:40:03.565222+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:40:13.356035+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 11001856
2019-10-23 08:40:15.306013+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:40:21.818764+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 13926400
2019-10-23 08:40:37.177306+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:40:49.129855+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:40:50.636354+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 851968
2019-10-23 08:40:53.380819+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 5484544
2019-10-23 08:41:01.489544+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:41:05.039429+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:41:19.146047+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 2650112
2019-10-23 08:41:27.038579+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:41:29.552880+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:41:34.502251+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:41:35.297931+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:41:40.992681+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:41:41.999290+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc growtime expired
2019-10-23 08:41:58.255714+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 7426272
2019-10-23 08:42:24.966011+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 7421952
2019-10-23 08:42:49.160744+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 11325440
2019-10-23 08:43:09.975501+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 2781184
2019-10-23 08:43:40.010961+0000 0x20117e   Default     0x0                  0      <zfs`arc_reclaim_thread> ZFS: arc_reclaim_thread, (old_)to_free has returned to zero from 268435456
lundman commented 4 years ago

Run spindump when it has gone unresponsive, so we can see all the stacks and where they are

whizzrd commented 4 years ago

Entire system is kind of unresponsive, spindump also locks up I was able to use Activity Monitor to sample the zfs and zpool processes ouputting something resembling a stack trace

Sampling process 28931 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling zfs (pid 28931) every 1 millisecond
Process:         zfs [28931]
Path:            /usr/local/bin/zfs
Load Address:    0x108444000
Identifier:      zfs
Version:         0
Code Type:       X86-64
Parent Process:  bash [7619]

Date/Time:       2019-10-23 11:14:42.411 +0200
Launch Time:     2019-10-23 10:47:38.510 +0200
OS Version:      Mac OS X 10.13.6 (17G8037)
Report Version:  7
Analysis Tool:   /usr/bin/sample

Physical footprint:         1160K
Physical footprint (peak):  1160K
----

Call graph:
    2775 Thread_3473545   DispatchQueue_1: com.apple.main-thread  (serial)
      2775 start  (in libdyld.dylib) + 1  [0x7fff66663015]
        2775 main  (in zfs) + 2008  [0x1084464ef]
          2775 zfs_do_list  (in zfs) + 677  [0x108449307]
            2775 zfs_for_each  (in zfs) + 460  [0x10844544b]
              2775 zfs_iter_root  (in libzfs.2.dylib) + 28  [0x108811fe3]
                2775 namespace_reload  (in libzfs.2.dylib) + 129  [0x108811cdf]
                  2775 zfs_ioctl  (in libzfs.2.dylib) + 55  [0x108838cd5]
                    2775 ioctl  (in libsystem_kernel.dylib) + 151  [0x7fff667ad4eb]
                      2775 __ioctl  (in libsystem_kernel.dylib) + 10  [0x7fff667b33fa]

Total number in stack (recursive counted multiple, when >=5):

Sort by top of stack, same collapsed (when >= 5):
        __ioctl  (in libsystem_kernel.dylib)        2775

Binary Images:
       0x108444000 -        0x108456fff +zfs (0) <C6E73E86-5916-3061-9077-6886EDF33F50> /usr/local/bin/zfs
       0x108469000 -        0x108474fff +libnvpair.1.dylib (0) <7CA2EACE-A919-35AE-866E-16E595376714> /usr/local/lib/libnvpair.1.dylib
       0x10848c000 -        0x10849fff7 +libuutil.1.dylib (0) <38177B5B-F4C7-3F6E-AA7E-42CE44437EE8> /usr/local/lib/libuutil.1.dylib
       0x1084be000 -        0x10862afe3 +libzpool.1.dylib (0) <0B6180AA-AF77-3DD4-B9D8-B11F389BFD49> /usr/local/lib/libzpool.1.dylib
       0x10880f000 -        0x108847ff7 +libzfs.2.dylib (0) <6CBC3221-8CE3-3431-91B4-027CA52574EA> /usr/local/lib/libzfs.2.dylib
       0x108866000 -        0x108868fff +libzfs_core.1.dylib (0) <36F1BC6D-4A68-30C3-8374-358577C3A2BC> /usr/local/lib/libzfs_core.1.dylib
       0x11478f000 -        0x1147d9acf  dyld (551.5) <30B355CB-35BA-3112-AA76-4E46CD45F699> /usr/lib/dyld
    0x7fff3a8f0000 -     0x7fff3a8f0fff  com.apple.Accelerate (1.11 - Accelerate 1.11) <2EF00EFE-06E8-3082-85C3-2988B422A19D> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
    0x7fff3a909000 -     0x7fff3b057ffb  com.apple.vImage (8.1 - ???) <56C275C1-459F-37CD-BF29-2E6D81C29E53> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
    0x7fff3b058000 -     0x7fff3b1f8ff3  libBLAS.dylib (1211.50.2) <93259222-2D94-333D-A1EB-09C6FC4A64B4> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
    0x7fff3b1f9000 -     0x7fff3b233fef  libBNNS.dylib (38.1) <0AAE0109-71D5-3B26-8401-3E62DDF97624> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib
    0x7fff3b234000 -     0x7fff3b60dff7  libLAPACK.dylib (1211.50.2) <70781D0B-AEAD-33EB-9900-10DC30F37D61> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
    0x7fff3b60e000 -     0x7fff3b624ff7  libLinearAlgebra.dylib (1211.50.2) <FB5A35EA-D636-3D69-AB53-B67E1EB8134C> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib
    0x7fff3b625000 -     0x7fff3b62aff3  libQuadrature.dylib (3) <3D6BF66A-55B2-3692-BAC7-DEB0C676ED29> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib
    0x7fff3b62b000 -     0x7fff3b6abfff  libSparse.dylib (79.50.2) <0DC25CDD-F8C1-3D6E-B472-8B060708424F> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparse.dylib
    0x7fff3b6ac000 -     0x7fff3b6beff7  libSparseBLAS.dylib (1211.50.2) <994A7D7E-FA3B-3943-A59C-8150B9AF86BE> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib
    0x7fff3b6bf000 -     0x7fff3b832fe7  libvDSP.dylib (622.50.5) <4BF310F4-31A3-3DA5-80E4-7F8014AD380B> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
    0x7fff3b833000 -     0x7fff3b8e9fef  libvMisc.dylib (622.50.5) <88143A88-4FDE-35A1-85FE-54FF0C2A9E43> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
    0x7fff3b8ea000 -     0x7fff3b8eafff  com.apple.Accelerate.vecLib (3.11 - vecLib 3.11) <880DEF40-389A-3F56-ACA1-B62A5BC57E59> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
    0x7fff3bbde000 -     0x7fff3ca3cfff  com.apple.AppKit (6.9 - 1561.61.100) <CC73A456-8FC5-38DA-B296-449A70FE3B3A> /System/Library/Frameworks/AppKit.framework/Versions/C/AppKit
    0x7fff3ca8e000 -     0x7fff3ca8efff  com.apple.ApplicationServices (48 - 50) <63B0C61F-CFE8-3884-853F-E6FB6CD84A44> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
    0x7fff3ca8f000 -     0x7fff3caf5fff  com.apple.ApplicationServices.ATS (377 - 445.5) <703CE7E4-426A-35C0-A229-F140F30F5340> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
    0x7fff3cb8e000 -     0x7fff3ccb0fff  libFontParser.dylib (222.1.7) <9C4CEFB4-0F08-3B60-BD50-0120A37F6EA2> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib
    0x7fff3ccb1000 -     0x7fff3ccfbff7  libFontRegistry.dylib (221.5) <8F68EA59-C8EE-3FA3-BD19-0F1A58441440> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
    0x7fff3ce3d000 -     0x7fff3ce41ff3  com.apple.ColorSyncLegacy (4.13.0 - 1) <A5FB2694-1559-34A8-A3D3-2029F68A63CA> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSyncLegacy.framework/Versions/A/ColorSyncLegacy
    0x7fff3cee1000 -     0x7fff3cf33ffb  com.apple.HIServices (1.22 - 625) <78A1FB4F-A39C-3159-B285-600C997335FB> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
    0x7fff3cf34000 -     0x7fff3cf42fff  com.apple.LangAnalysis (1.7.0 - 1.7.0) <B65FF7E6-E9B5-34D8-8CA7-63D415A8A9A6> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/LangAnalysis.framework/Versions/A/LangAnalysis
    0x7fff3cf43000 -     0x7fff3cf8ffff  com.apple.print.framework.PrintCore (13.4 - 503.2) <B90C67C1-0292-3CEC-885D-F1882CD104BE> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/PrintCore.framework/Versions/A/PrintCore
    0x7fff3cf90000 -     0x7fff3cfcafff  com.apple.QD (3.12 - 404.2) <38B20AFF-9D54-3B52-A6DC-C0D71380AA5F> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/QD.framework/Versions/A/QD
    0x7fff3cfcb000 -     0x7fff3cfd7fff  com.apple.speech.synthesis.framework (7.8.1 - 7.8.1) <A08DE016-C8F2-3B0E-BD34-15959D13DBF0> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
    0x7fff3cfd8000 -     0x7fff3d266ff7  com.apple.audio.toolbox.AudioToolbox (1.14 - 1.14) <E0B8B5D8-80A0-308B-ABD6-F8612102B5D8> /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
    0x7fff3d268000 -     0x7fff3d268fff  com.apple.audio.units.AudioUnit (1.14 - 1.14) <ABF8778E-4F9D-305E-A528-DE406A1A2B68> /System/Library/Frameworks/AudioUnit.framework/Versions/A/AudioUnit
    0x7fff3d58b000 -     0x7fff3d926ff7  com.apple.CFNetwork (902.3.1 - 902.3.1) <4C012538-BB8A-32F3-AACA-011092BEFAB1> /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
    0x7fff3d940000 -     0x7fff3dc45fff  com.apple.HIToolbox (2.1.1 - 911.10) <BF7F9C0E-C732-3FB2-9BBC-362888BDA57B> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/HIToolbox.framework/Versions/A/HIToolbox
    0x7fff3dc50000 -     0x7fff3dce5ffb  com.apple.ink.framework (10.9 - 221) <5206C8B0-22DA-36C9-998E-846EDB626D5B> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/Ink.framework/Versions/A/Ink
    0x7fff3dd26000 -     0x7fff3dd2cfff  com.apple.speech.recognition.framework (6.0.3 - 6.0.3) <2ED8643D-B0C3-3F17-82A2-BBF13E6CBABC> /System/Library/Frameworks/Carbon.framework/Versions/A/Frameworks/SpeechRecognition.framework/Versions/A/SpeechRecognition
    0x7fff3de5b000 -     0x7fff3df14fff  com.apple.ColorSync (4.13.0 - 3325) <D283C285-447D-3258-A7E4-59532123B8FF> /System/Library/Frameworks/ColorSync.framework/Versions/A/ColorSync
    0x7fff3e0a1000 -     0x7fff3e134ff7  com.apple.audio.CoreAudio (4.3.0 - 4.3.0) <EB35D3EC-56EA-33E6-98DC-BDC3A5FA8ACE> /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
    0x7fff3e19b000 -     0x7fff3e1c4ffb  com.apple.CoreBluetooth (1.0 - 1) <E1335074-9D07-370E-8440-61C4874BAC56> /System/Library/Frameworks/CoreBluetooth.framework/Versions/A/CoreBluetooth
    0x7fff3e1c5000 -     0x7fff3e529ff3  com.apple.CoreData (120 - 851) <8D605466-8676-3D45-9A4B-8980971551C3> /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
    0x7fff3e52a000 -     0x7fff3e60dfff  com.apple.CoreDisplay (99.14 - 99.14) <A1B91ADD-828D-33A0-8A92-CC3F83DF89D0> /System/Library/Frameworks/CoreDisplay.framework/Versions/A/CoreDisplay
    0x7fff3e60e000 -     0x7fff3eaaafff  com.apple.CoreFoundation (6.9 - 1455.11) <EA9F0BF7-6A1F-36C9-A42E-6DE3DBCD99BD> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    0x7fff3eaac000 -     0x7fff3f0daff7  com.apple.CoreGraphics (2.0 - 1161.21.2) <703FBD9C-F88A-3820-A322-49B9C765715D> /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
    0x7fff3f0dc000 -     0x7fff3f3cbfff  com.apple.CoreImage (13.0.0 - 579.5) <AAE2DFD0-9B0A-3D56-8A3E-C460BAF70394> /System/Library/Frameworks/CoreImage.framework/Versions/A/CoreImage
    0x7fff3f7a0000 -     0x7fff3f7a0fff  com.apple.CoreServices (822.37 - 822.37) <C7BB281A-16A2-321A-90F4-B970278D9C2A> /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
    0x7fff3f7a1000 -     0x7fff3f815ffb  com.apple.AE (735.2 - 735.2) <690A6F47-1306-371F-B257-783D8F83EA9E> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
    0x7fff3f816000 -     0x7fff3faedfff  com.apple.CoreServices.CarbonCore (1178.4.2 - 1178.4.2) <08AF7B7B-219B-3957-86CE-C96B423C823C> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
    0x7fff3faee000 -     0x7fff3fb22fff  com.apple.DictionaryServices (1.2 - 284.2) <6505B075-41C3-3C62-A4C3-85CE3F6825CD> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
    0x7fff3fb23000 -     0x7fff3fb2bffb  com.apple.CoreServices.FSEvents (1239.50.1 - 1239.50.1) <3637CEC7-DF0E-320E-9634-44A442925C65> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/FSEvents.framework/Versions/A/FSEvents
    0x7fff3fb2c000 -     0x7fff3fceaff7  com.apple.LaunchServices (822.37 - 822.37) <6AA93307-220A-3417-BBC2-FE1C0DE0B652> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
    0x7fff3fceb000 -     0x7fff3fd9bfff  com.apple.Metadata (10.7.0 - 1191.7) <3DD530A7-E104-3469-98BB-20449834B2FE> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
    0x7fff3fd9c000 -     0x7fff3fdfcfff  com.apple.CoreServices.OSServices (822.37 - 822.37) <4AD2FC98-C6CB-392A-A22F-196A723D7FAE> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
    0x7fff3fdfd000 -     0x7fff3fe6bfff  com.apple.SearchKit (1.4.0 - 1.4.0) <3662545A-B1CF-3079-BDCD-C83855CEFEEE> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
    0x7fff3fe6c000 -     0x7fff3fe90ffb  com.apple.coreservices.SharedFileList (71.21 - 71.21) <1B5228EF-D869-3A50-A373-7F4B0289FADD> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
    0x7fff40131000 -     0x7fff40281fff  com.apple.CoreText (352.0 - 578.22) <6129F39D-284D-3BBF-8999-7854AB61C01C> /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
    0x7fff40282000 -     0x7fff402bcfff  com.apple.CoreVideo (1.8 - 0.0) <86CCC036-51BB-3DD1-9601-D93798BCCD0F> /System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo
    0x7fff402bd000 -     0x7fff40348ff3  com.apple.framework.CoreWLAN (13.0 - 1350.2) <DC9E1B0B-FD96-38A7-A6E0-820CA414AA94> /System/Library/Frameworks/CoreWLAN.framework/Versions/A/CoreWLAN
    0x7fff405c3000 -     0x7fff405c8fff  com.apple.DiskArbitration (2.7 - 2.7) <2F628584-8177-3894-92CE-9CA6089B61C6> /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
    0x7fff40789000 -     0x7fff40b50ffb  com.apple.Foundation (6.9 - 1455.11) <121FC177-8542-36BA-9F72-CCEAE8AF88FE> /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
    0x7fff40bc1000 -     0x7fff40bf1ff3  com.apple.GSS (4.0 - 2.0) <FC69A120-3C19-3802-87B4-C29217758143> /System/Library/Frameworks/GSS.framework/Versions/A/GSS
    0x7fff40d03000 -     0x7fff40e07ffb  com.apple.Bluetooth (6.0.7 - 6.0.7f15) <6B622256-2E03-3055-B28A-803A001C9912> /System/Library/Frameworks/IOBluetooth.framework/Versions/A/IOBluetooth
    0x7fff40e67000 -     0x7fff40f02fff  com.apple.framework.IOKit (2.0.2 - 1445.71.6) <C68CE7ED-85CA-3D1F-BAA0-908327BEC0DE> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
    0x7fff40f04000 -     0x7fff40f0bfff  com.apple.IOSurface (211.15 - 211.15) <9FD406F1-6BF2-35B0-8339-DF83A1A661EB> /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface
    0x7fff40f62000 -     0x7fff410dffff  com.apple.ImageIO.framework (3.3.0 - 1739.3.1) <3564ADD2-803F-35CC-AB7F-6F41241010FF> /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
    0x7fff410e0000 -     0x7fff410e4ffb  libGIF.dylib (1739.3.1) <3CB9E6D5-FF76-37E5-921D-57F2C36D8CFC> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
    0x7fff410e5000 -     0x7fff411ccff7  libJP2.dylib (1739.3.1) <2B233EA0-59CC-3D7D-81AF-F471448F5271> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
    0x7fff411cd000 -     0x7fff411f0fff  libJPEG.dylib (1739.3.1) <4B3C1085-9177-3872-BCAF-965CBCD6A906> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
    0x7fff414cd000 -     0x7fff414f3ff3  libPng.dylib (1739.3.1) <734D972E-0BD6-3DAC-A465-43A8AF9E0BC8> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
    0x7fff414f4000 -     0x7fff414f6ff3  libRadiance.dylib (1739.3.1) <99967613-358F-377D-BEC2-C52CC2B18811> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
    0x7fff414f7000 -     0x7fff41544ff3  libTIFF.dylib (1739.3.1) <BEAC3497-7D7F-3A09-B232-07E9792ACF88> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
    0x7fff423fe000 -     0x7fff42417ff7  com.apple.Kerberos (3.0 - 1) <F86DCCDF-93C1-38B3-82C2-477C12E8EE6D> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
    0x7fff42df9000 -     0x7fff42e7afff  com.apple.Metal (125.30 - 125.30) <6B9EBDEE-C64C-3C1C-922A-0363B642C9BC> /System/Library/Frameworks/Metal.framework/Versions/A/Metal
    0x7fff42e97000 -     0x7fff42eb2fff  com.apple.MetalPerformanceShaders.MPSCore (1.0 - 1) <AD754E8F-CA00-3878-9AF3-208C224A230B> /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSCore.framework/Versions/A/MPSCore
    0x7fff42eb3000 -     0x7fff42f22fef  com.apple.MetalPerformanceShaders.MPSImage (1.0 - 1) <338B7779-E608-3D68-8A07-2ACC11299744> /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSImage.framework/Versions/A/MPSImage
    0x7fff42f23000 -     0x7fff42f47fff  com.apple.MetalPerformanceShaders.MPSMatrix (1.0 - 1) <9CE072D7-853B-3939-9645-7EB951376B87> /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSMatrix.framework/Versions/A/MPSMatrix
    0x7fff42f48000 -     0x7fff4302fff7  com.apple.MetalPerformanceShaders.MPSNeuralNetwork (1.0 - 1) <0DE891AD-27E5-38FF-AEC8-4A95356C4357> /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSNeuralNetwork.framework/Versions/A/MPSNeuralNetwork
    0x7fff43030000 -     0x7fff43030ff7  com.apple.MetalPerformanceShaders.MetalPerformanceShaders (1.0 - 1) <2D2D261C-50B0-32F9-BF9A-5C01382BB528> /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/MetalPerformanceShaders
    0x7fff4402f000 -     0x7fff4403bffb  com.apple.NetFS (6.0 - 4.0) <471DD96F-FA2E-3FE9-9746-2519A6780D1A> /System/Library/Frameworks/NetFS.framework/Versions/A/NetFS
    0x7fff46e2d000 -     0x7fff46e87ff7  com.apple.opencl (2.8.24 - 2.8.24) <CF922AFD-CBA4-3B23-A0C1-E3E5362BB6C7> /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL
    0x7fff46e88000 -     0x7fff46ea4ffb  com.apple.CFOpenDirectory (10.13 - 207.50.1) <29F55F7B-379F-3053-8FF3-5C6675A3DD4D> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
    0x7fff46ea5000 -     0x7fff46eb0fff  com.apple.OpenDirectory (10.13 - 207.50.1) <F895547D-4915-353F-9C1E-E95172BA803B> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory
    0x7fff4802f000 -     0x7fff48031fff  libCVMSPluginSupport.dylib (16.7.4) <6846D939-478C-3AE3-9DCD-CD4F4CD05409> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
    0x7fff48032000 -     0x7fff48037ffb  libCoreFSCache.dylib (162.9) <DD9DD721-2957-3D05-B361-70AF9EBFB280> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreFSCache.dylib
    0x7fff48038000 -     0x7fff4803cfff  libCoreVMClient.dylib (162.9) <4E4F3EA2-5B53-31A3-8256-54EFAA94E4D6> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
    0x7fff4803d000 -     0x7fff48046ff3  libGFXShared.dylib (16.7.4) <6187AD8B-1CD2-3775-84CD-76A0EA8BCDD4> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
    0x7fff48047000 -     0x7fff48052fff  libGL.dylib (16.7.4) <2BB333D3-5C61-33DF-8545-06DF2D08B83D> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
    0x7fff48053000 -     0x7fff4808efe7  libGLImage.dylib (16.7.4) <4DA003CE-0B74-3FE4-808C-B2FBCE517EB4> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
    0x7fff481fd000 -     0x7fff4823bffb  libGLU.dylib (16.7.4) <BCB09CD8-EB0E-38FA-8B5A-9E29532EE364> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
    0x7fff48bb3000 -     0x7fff48bc2ff3  com.apple.opengl (16.7.4 - 16.7.4) <7B71A596-B6D8-335A-B7CA-8684D83B9C0E> /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
    0x7fff49a11000 -     0x7fff49c5dff7  com.apple.QuartzCore (1.11 - 584.64.2) <DDA39332-DE09-3737-A332-0002AE05DD39> /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
    0x7fff4a495000 -     0x7fff4a7c0fff  com.apple.security (7.0 - 58286.70.14) <CAAA4591-34B7-3281-97B9-0D4BCFBB7816> /System/Library/Frameworks/Security.framework/Versions/A/Security
    0x7fff4a7c1000 -     0x7fff4a84dff7  com.apple.securityfoundation (6.0 - 55185.50.5) <B5F933AE-BF0B-36CB-B2D4-3C07DD952BCE> /System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
    0x7fff4a87f000 -     0x7fff4a883ffb  com.apple.xpc.ServiceManagement (1.0 - 1) <E3146BBE-7F32-3605-AD2B-54C0E70759DC> /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement
    0x7fff4ac28000 -     0x7fff4ac98ff3  com.apple.SystemConfiguration (1.17 - 1.17) <CD4A4C66-7886-3323-A3B9-0C8952500856> /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
    0x7fff4db95000 -     0x7fff4dc28fff  com.apple.APFS (1.0 - 1) <7CC77BF1-881B-357A-9E05-A73C661F830E> /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
    0x7fff4e855000 -     0x7fff4e87dfff  com.apple.framework.Apple80211 (13.0 - 1370.4) <B37A91A0-5823-3F97-B575-9BB804FCEB00> /System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Apple80211
    0x7fff4e87f000 -     0x7fff4e88efef  com.apple.AppleFSCompression (96.60.1 - 1.0) <A7C875C4-F5EE-3272-AFB6-57C9FD5352B3> /System/Library/PrivateFrameworks/AppleFSCompression.framework/Versions/A/AppleFSCompression
    0x7fff4e98d000 -     0x7fff4e998ff7  com.apple.AppleIDAuthSupport (1.0 - 1) <2FAF5567-CDB3-33EF-AB71-05D37F2248B7> /System/Library/PrivateFrameworks/AppleIDAuthSupport.framework/Versions/A/AppleIDAuthSupport
    0x7fff4e9d2000 -     0x7fff4ea1aff3  com.apple.AppleJPEG (1.0 - 1) <8DD410CB-76A1-3F22-9A9F-0491FA0CEB4A> /System/Library/PrivateFrameworks/AppleJPEG.framework/Versions/A/AppleJPEG
    0x7fff4ea55000 -     0x7fff4ea7dfff  com.apple.applesauce (1.0 - ???) <CCA8B094-1BCE-3AE3-A0A7-D544C818DE36> /System/Library/PrivateFrameworks/AppleSauce.framework/Versions/A/AppleSauce
    0x7fff4eed0000 -     0x7fff4f169ffb  com.apple.AuthKit (1.0 - 1) <6CA71A11-91C5-307C-B933-9FCDEDCB580A> /System/Library/PrivateFrameworks/AuthKit.framework/Versions/A/AuthKit
    0x7fff4f2a0000 -     0x7fff4f2a7ff7  com.apple.coreservices.BackgroundTaskManagement (1.0 - 57.1) <51A41CA3-DB1D-3380-993E-99C54AEE518E> /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
    0x7fff4f2a8000 -     0x7fff4f32fff7  com.apple.backup.framework (1.9.6 - 1.9.6) <3C96FD26-C7F2-3F37-885A-5A71372FA8F4> /System/Library/PrivateFrameworks/Backup.framework/Versions/A/Backup
    0x7fff50ce9000 -     0x7fff50cf2ff3  com.apple.CommonAuth (4.0 - 2.0) <21AF4AC2-E650-35F9-AB33-3EBA2769FBC0> /System/Library/PrivateFrameworks/CommonAuth.framework/Versions/A/CommonAuth
    0x7fff51806000 -     0x7fff51816ff7  com.apple.CoreEmoji (1.0 - 69.3) <A4357F5C-0C38-3A61-B456-D7321EB2CEE5> /System/Library/PrivateFrameworks/CoreEmoji.framework/Versions/A/CoreEmoji
    0x7fff51faa000 -     0x7fff51fb2ff3  com.apple.CorePhoneNumbers (1.0 - 1) <A5D41251-9F38-3AB9-9DE7-F77023FAAA44> /System/Library/PrivateFrameworks/CorePhoneNumbers.framework/Versions/A/CorePhoneNumbers
    0x7fff524e8000 -     0x7fff52579fff  com.apple.CoreSymbolication (9.3 - 64026.2) <A8D4315F-5DD5-3164-8672-ECDAF2766644> /System/Library/PrivateFrameworks/CoreSymbolication.framework/Versions/A/CoreSymbolication
    0x7fff525fc000 -     0x7fff52731fff  com.apple.coreui (2.1 - 494.1) <B2C515C3-FCE8-3B28-A225-05AD917F509B> /System/Library/PrivateFrameworks/CoreUI.framework/Versions/A/CoreUI
    0x7fff52732000 -     0x7fff52863fff  com.apple.CoreUtils (5.6 - 560.11) <1A02D6F0-8C65-3FAE-AD63-56477EDE4773> /System/Library/PrivateFrameworks/CoreUtils.framework/Versions/A/CoreUtils
    0x7fff528b8000 -     0x7fff5291cfff  com.apple.framework.CoreWiFi (13.0 - 1350.2) <50E8598A-0611-323D-B9A8-72A336424090> /System/Library/PrivateFrameworks/CoreWiFi.framework/Versions/A/CoreWiFi
    0x7fff5291d000 -     0x7fff5292dff7  com.apple.CrashReporterSupport (10.13 - 1) <A909F468-0648-3F51-A77E-3F9ADBC9A941> /System/Library/PrivateFrameworks/CrashReporterSupport.framework/Versions/A/CrashReporterSupport
    0x7fff529ac000 -     0x7fff529bbff7  com.apple.framework.DFRFoundation (1.0 - 191.7) <3B8ED6F7-5DFF-34C3-BA90-DDB85679684C> /System/Library/PrivateFrameworks/DFRFoundation.framework/Versions/A/DFRFoundation
    0x7fff529be000 -     0x7fff529c2ffb  com.apple.DSExternalDisplay (3.1 - 380) <901B7F6D-376A-3848-99D0-170C4D00F776> /System/Library/PrivateFrameworks/DSExternalDisplay.framework/Versions/A/DSExternalDisplay
    0x7fff52a44000 -     0x7fff52abafff  com.apple.datadetectorscore (7.0 - 590.3) <B4706195-CBE6-320D-A0E1-A9D4BDF52791> /System/Library/PrivateFrameworks/DataDetectorsCore.framework/Versions/A/DataDetectorsCore
    0x7fff52b08000 -     0x7fff52b48ff7  com.apple.DebugSymbols (181.0 - 181.0) <299A0238-ED78-3676-B131-274D972824AA> /System/Library/PrivateFrameworks/DebugSymbols.framework/Versions/A/DebugSymbols
    0x7fff52b49000 -     0x7fff52c78fff  com.apple.desktopservices (1.12.5 - 1.12.5) <7739C9A5-64D9-31A5-899B-5FFA242AD70D> /System/Library/PrivateFrameworks/DesktopServicesPriv.framework/Versions/A/DesktopServicesPriv
    0x7fff53a8d000 -     0x7fff53ebbfff  com.apple.vision.FaceCore (3.3.2 - 3.3.2) <B574FE33-4A41-3611-9738-388EBAF03E37> /System/Library/PrivateFrameworks/FaceCore.framework/Versions/A/FaceCore
    0x7fff57183000 -     0x7fff57188fff  com.apple.GPUWrangler (3.20.16 - 3.20.16) <C24B2DA4-998E-3880-9C73-EB93AC04C357> /System/Library/PrivateFrameworks/GPUWrangler.framework/Versions/A/GPUWrangler
    0x7fff57efe000 -     0x7fff57f0dfff  com.apple.GraphVisualizer (1.0 - 5) <B993B8A2-5700-3DFC-9EB7-4CCEE8F959F1> /System/Library/PrivateFrameworks/GraphVisualizer.framework/Versions/A/GraphVisualizer
    0x7fff57f90000 -     0x7fff58004fff  com.apple.Heimdal (4.0 - 2.0) <BAE3D978-2285-35DF-907E-02BC37986D02> /System/Library/PrivateFrameworks/Heimdal.framework/Versions/A/Heimdal
    0x7fff58917000 -     0x7fff5891eff7  com.apple.IOAccelerator (378.28 - 378.28) <BE5CBEAA-314E-348B-BE85-7E4B2E4C42FF> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
    0x7fff58922000 -     0x7fff58939fff  com.apple.IOPresentment (1.0 - 35.1) <53B47F2A-9A9A-33F5-ADCB-FF766292B93A> /System/Library/PrivateFrameworks/IOPresentment.framework/Versions/A/IOPresentment
    0x7fff58d04000 -     0x7fff58d2affb  com.apple.IconServices (97.6 - 97.6) <A56D826D-20D2-34BE-AACC-A80CFCB4E915> /System/Library/PrivateFrameworks/IconServices.framework/Versions/A/IconServices
    0x7fff58faf000 -     0x7fff58fc2ff3  com.apple.security.KeychainCircle.KeychainCircle (1.0 - 1) <CA865016-5555-3522-841D-42D974AAB981> /System/Library/PrivateFrameworks/KeychainCircle.framework/Versions/A/KeychainCircle
    0x7fff58fc3000 -     0x7fff590b8ff7  com.apple.LanguageModeling (1.0 - 159.5.3) <7F0AC200-E3DD-39FB-8A95-00DD70B66A9F> /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
    0x7fff590b9000 -     0x7fff590fbfff  com.apple.Lexicon-framework (1.0 - 33.5) <DC94CF9E-1EB4-3C0E-B298-CA1190885276> /System/Library/PrivateFrameworks/Lexicon.framework/Versions/A/Lexicon
    0x7fff590ff000 -     0x7fff59106ff7  com.apple.LinguisticData (1.0 - 238.3) <49A54649-1021-3DBD-99B8-1B2EDFFA5378> /System/Library/PrivateFrameworks/LinguisticData.framework/Versions/A/LinguisticData
    0x7fff59e19000 -     0x7fff59e82ff7  com.apple.gpusw.MetalTools (1.0 - 1) <B4217182-B2AA-3AA3-80E8-F4C80B71BDDC> /System/Library/PrivateFrameworks/MetalTools.framework/Versions/A/MetalTools
    0x7fff5a012000 -     0x7fff5a02bfff  com.apple.MobileKeyBag (2.0 - 1.0) <7D657BC6-B779-3E51-B50D-07C95791DD04> /System/Library/PrivateFrameworks/MobileKeyBag.framework/Versions/A/MobileKeyBag
    0x7fff5a0b7000 -     0x7fff5a0e1ffb  com.apple.MultitouchSupport.framework (1614.1 - 1614.1) <A0709B43-FA9E-3617-8E7A-D68CDFAB2167> /System/Library/PrivateFrameworks/MultitouchSupport.framework/Versions/A/MultitouchSupport
    0x7fff5a348000 -     0x7fff5a353fff  com.apple.NetAuth (6.2 - 6.2) <2CFA1228-61A1-3708-988D-30499A1D6A9F> /System/Library/PrivateFrameworks/NetAuth.framework/Versions/A/NetAuth
    0x7fff5bbec000 -     0x7fff5bbfcffb  com.apple.PerformanceAnalysis (1.194 - 194) <8022ED1F-CE70-388E-B37B-6FB1E5F90CF2> /System/Library/PrivateFrameworks/PerformanceAnalysis.framework/Versions/A/PerformanceAnalysis
    0x7fff5d9bb000 -     0x7fff5d9d9fff  com.apple.ProtocolBuffer (1 - 260) <40704740-4A53-3010-A49B-08D1D69D1D5E> /System/Library/PrivateFrameworks/ProtocolBuffer.framework/Versions/A/ProtocolBuffer
    0x7fff5dbb4000 -     0x7fff5dbd7ffb  com.apple.RemoteViewServices (2.0 - 125) <592323D1-CB44-35F1-9921-4C2AB8D920A0> /System/Library/PrivateFrameworks/RemoteViewServices.framework/Versions/A/RemoteViewServices
    0x7fff5f4fd000 -     0x7fff5f612ff7  com.apple.Sharing (1050.22.3 - 1050.22.3) <0E73D486-D46A-3E6C-BB6C-49FC6CF1A0B6> /System/Library/PrivateFrameworks/Sharing.framework/Versions/A/Sharing
    0x7fff5f63d000 -     0x7fff5f63eff7  com.apple.performance.SignpostNotification (1.2.6 - 2.6) <8F04800F-3570-3392-A24D-B229FF03F7F9> /System/Library/PrivateFrameworks/SignpostNotification.framework/Versions/A/SignpostNotification
    0x7fff6039b000 -     0x7fff60637fff  com.apple.SkyLight (1.600.0 - 312.103.11) <2E04D36B-F539-356A-BED5-781CD98C2C96> /System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/SkyLight
    0x7fff60e00000 -     0x7fff60e0dfff  com.apple.SpeechRecognitionCore (4.6.1 - 4.6.1) <87EE7AB5-6925-3D21-BE00-F155CB457699> /System/Library/PrivateFrameworks/SpeechRecognitionCore.framework/Versions/A/SpeechRecognitionCore
    0x7fff619b3000 -     0x7fff61a3cfc7  com.apple.Symbolication (9.3 - 64033) <FAA17252-6378-34A4-BBBB-22DF54EC1626> /System/Library/PrivateFrameworks/Symbolication.framework/Versions/A/Symbolication
    0x7fff61fac000 -     0x7fff61fb4ff7  com.apple.TCC (1.0 - 1) <E1EB7272-FE6F-39AB-83CA-B2B5F2A88D9B> /System/Library/PrivateFrameworks/TCC.framework/Versions/A/TCC
    0x7fff621c1000 -     0x7fff6227eff7  com.apple.TextureIO (3.7 - 3.7) <F8BAC954-405D-3CC3-AB7B-048C866EF980> /System/Library/PrivateFrameworks/TextureIO.framework/Versions/A/TextureIO
    0x7fff6232f000 -     0x7fff624dfff3  com.apple.UIFoundation (1.0 - 547.6) <F9D3CB20-8567-30A1-9A96-E61399B6CAAE> /System/Library/PrivateFrameworks/UIFoundation.framework/Versions/A/UIFoundation
    0x7fff63c5b000 -     0x7fff63c5dffb  com.apple.loginsupport (1.0 - 1) <D1232C1B-80EA-3DF8-9466-013695D0846E> /System/Library/PrivateFrameworks/login.framework/Versions/A/Frameworks/loginsupport.framework/Versions/A/loginsupport
    0x7fff63dc4000 -     0x7fff63df7ff7  libclosured.dylib (551.5) <112BC241-6626-3848-8DD8-B34B5B6F7ABC> /usr/lib/closure/libclosured.dylib
    0x7fff63eb1000 -     0x7fff63eeaff7  libCRFSuite.dylib (41) <FE5EDB68-2593-3C2E-BBAF-1C52D206F296> /usr/lib/libCRFSuite.dylib
    0x7fff63eeb000 -     0x7fff63ef6fff  libChineseTokenizer.dylib (28) <53633C9B-A3A8-36F7-A53C-432D802F4BB8> /usr/lib/libChineseTokenizer.dylib
    0x7fff63f88000 -     0x7fff63f89ff3  libDiagnosticMessagesClient.dylib (104) <9712E980-76EE-3A89-AEA6-DF4BAF5C0574> /usr/lib/libDiagnosticMessagesClient.dylib
    0x7fff63fc0000 -     0x7fff6418aff3  libFosl_dynamic.dylib (17.8) <C58ED77A-4986-31C2-994C-34DDFB8106F0> /usr/lib/libFosl_dynamic.dylib
    0x7fff641c2000 -     0x7fff641c2fff  libOpenScriptingUtil.dylib (174) <610F0242-7CE5-3C86-951B-B646562694AF> /usr/lib/libOpenScriptingUtil.dylib
    0x7fff642f9000 -     0x7fff642fdffb  libScreenReader.dylib (562.18.5) <0FEFDAC8-0FDF-38A0-AB88-A84E0DBA058F> /usr/lib/libScreenReader.dylib
    0x7fff642fe000 -     0x7fff642ffffb  libSystem.B.dylib (1252.50.4) <C084DB19-08B8-322B-9344-67F2970929DA> /usr/lib/libSystem.B.dylib
    0x7fff64392000 -     0x7fff64392fff  libapple_crypto.dylib (109.50.14) <48BA2E76-BF2F-3522-A54E-D7FB7EAF7A57> /usr/lib/libapple_crypto.dylib
    0x7fff64393000 -     0x7fff643a9ff7  libapple_nghttp2.dylib (1.24) <01402BC4-4822-3676-9C80-50D83F816424> /usr/lib/libapple_nghttp2.dylib
    0x7fff643aa000 -     0x7fff643d4ff3  libarchive.2.dylib (54) <8FC28DD8-E315-3C3E-95FE-D1D2CBE49888> /usr/lib/libarchive.2.dylib
    0x7fff643d5000 -     0x7fff644d8fe7  libate.dylib (1.13.1) <DABEA1B1-806C-34C9-8AFF-DEB6AB2829E3> /usr/lib/libate.dylib
    0x7fff644dc000 -     0x7fff644dcff3  libauto.dylib (187) <A05C7900-F8C7-3E75-8D3F-909B40C19717> /usr/lib/libauto.dylib
    0x7fff644dd000 -     0x7fff64595ff3  libboringssl.dylib (109.50.14) <E6813F87-B5E4-3F7F-A725-E6A7F2BD02EC> /usr/lib/libboringssl.dylib
    0x7fff64596000 -     0x7fff645a6ff3  libbsm.0.dylib (39) <6BC96A72-AFBE-34FD-91B1-748A530D8AE6> /usr/lib/libbsm.0.dylib
    0x7fff645a7000 -     0x7fff645b4ffb  libbz2.1.0.dylib (38) <0A5086BB-4724-3C14-979D-5AD4F26B5B45> /usr/lib/libbz2.1.0.dylib
    0x7fff645b5000 -     0x7fff6460bfff  libc++.1.dylib (400.9) <7D3DACCC-3804-393C-ABC1-1A580FD00CB6> /usr/lib/libc++.1.dylib
    0x7fff6460c000 -     0x7fff64630ff7  libc++abi.dylib (400.8.2) <EF5E37D7-11D9-3530-BE45-B986612D13E2> /usr/lib/libc++abi.dylib
    0x7fff64632000 -     0x7fff64642fff  libcmph.dylib (6) <890DEC4C-4334-393C-8B56-7C8560BBED9D> /usr/lib/libcmph.dylib
    0x7fff64643000 -     0x7fff6465afd3  libcompression.dylib (47.60.2) <CC7ACB0A-49CD-3564-978F-49F843186F27> /usr/lib/libcompression.dylib
    0x7fff64905000 -     0x7fff6491dff7  libcoretls.dylib (155.50.1) <D350052E-DC4D-3185-ADBA-BA48EDCEE955> /usr/lib/libcoretls.dylib
    0x7fff6491e000 -     0x7fff6491fff3  libcoretls_cfhelpers.dylib (155.50.1) <B297F5D8-F2FE-3566-A752-E9D998B9C039> /usr/lib/libcoretls_cfhelpers.dylib
    0x7fff64df0000 -     0x7fff64e45ff3  libcups.2.dylib (462.2.7) <46179788-4363-38B2-9204-E8AC8123D899> /usr/lib/libcups.2.dylib
    0x7fff64f85000 -     0x7fff64f85fff  libenergytrace.dylib (16) <A92AB8B8-B986-3CE6-980D-D55090FEF387> /usr/lib/libenergytrace.dylib
    0x7fff64fbc000 -     0x7fff64fc1ff3  libheimdal-asn1.dylib (520.50.9) <EF65C17E-31CD-31DC-876C-BF3EF393099C> /usr/lib/libheimdal-asn1.dylib
    0x7fff64fed000 -     0x7fff650deff7  libiconv.2.dylib (51.50.1) <2FEC9707-3FAF-3828-A50D-8605086D060F> /usr/lib/libiconv.2.dylib
    0x7fff650df000 -     0x7fff65306ffb  libicucore.A.dylib (59181.0.1) <2CE6205F-D375-3BF5-AA0A-3254BC4773D0> /usr/lib/libicucore.A.dylib
    0x7fff65353000 -     0x7fff65354fff  liblangid.dylib (128) <39C39393-0D05-301D-93B2-F224FC4949AA> /usr/lib/liblangid.dylib
    0x7fff65355000 -     0x7fff6536effb  liblzma.5.dylib (10) <3D419A50-961F-37D2-8A01-3DC7AB7B8D18> /usr/lib/liblzma.5.dylib
    0x7fff6536f000 -     0x7fff65385ff7  libmarisa.dylib (9) <D6D2D55D-1D2E-3442-B152-B18803C0ABB4> /usr/lib/libmarisa.dylib
    0x7fff65436000 -     0x7fff6565eff7  libmecabra.dylib (779.7.6) <F462F170-E872-3D09-B219-973D5E99C09F> /usr/lib/libmecabra.dylib
    0x7fff65836000 -     0x7fff659b1fff  libnetwork.dylib (1229.70.2) <E185D902-AC7F-3044-87C0-AE2887C59CE7> /usr/lib/libnetwork.dylib
    0x7fff65a38000 -     0x7fff65e293b7  libobjc.A.dylib (723) <BEEA51E2-1C4D-3672-87F6-F267B4AC6221> /usr/lib/libobjc.A.dylib
    0x7fff65e3c000 -     0x7fff65e40fff  libpam.2.dylib (22) <7B4D2CE2-1438-387A-9802-5CEEFBF26F86> /usr/lib/libpam.2.dylib
    0x7fff65e43000 -     0x7fff65e77fff  libpcap.A.dylib (79.20.1) <FA13918B-A247-3181-B256-9B852C7BA316> /usr/lib/libpcap.A.dylib
    0x7fff65ef6000 -     0x7fff65f12ffb  libresolv.9.dylib (65) <E8F3415B-4472-3202-8901-41FD87981DB2> /usr/lib/libresolv.9.dylib
    0x7fff65f63000 -     0x7fff660f0ff7  libsqlite3.dylib (274.8.1) <B70A2908-E0F7-364E-952A-1D2D19787B89> /usr/lib/libsqlite3.dylib
    0x7fff662c4000 -     0x7fff66324ff3  libusrtcp.dylib (1229.70.2) <1E065228-D0E3-3808-9405-894056C6BEC0> /usr/lib/libusrtcp.dylib
    0x7fff66325000 -     0x7fff66328ffb  libutil.dylib (51.20.1) <216D18E5-0BAF-3EAF-A38E-F6AC37CBABD9> /usr/lib/libutil.dylib
    0x7fff66329000 -     0x7fff66336ff3  libxar.1.dylib (417.1) <351781D5-3CCF-320F-BB5A-4DFCC1CCD31D> /usr/lib/libxar.1.dylib
    0x7fff6633a000 -     0x7fff66421fff  libxml2.2.dylib (31.14) <4B6055E6-D8AD-342F-BCD7-F2EDD9069014> /usr/lib/libxml2.2.dylib
    0x7fff66422000 -     0x7fff6644afff  libxslt.1.dylib (15.14) <AECAB47A-B160-3D6D-BDDC-24B05C6EB8D4> /usr/lib/libxslt.1.dylib
    0x7fff6644b000 -     0x7fff6645dffb  libz.1.dylib (70) <48C67CFC-940D-3857-8DAD-857774605352> /usr/lib/libz.1.dylib
    0x7fff664f9000 -     0x7fff664fdff7  libcache.dylib (80) <092479CB-1008-3A83-BECF-E115F24D13C1> /usr/lib/system/libcache.dylib
    0x7fff664fe000 -     0x7fff66508ff3  libcommonCrypto.dylib (60118.50.1) <029F5985-9B6E-3DCB-9B96-FD007678C6A7> /usr/lib/system/libcommonCrypto.dylib
    0x7fff66509000 -     0x7fff66510fff  libcompiler_rt.dylib (62) <968B8E3F-3681-3230-9D78-BB8732024F6E> /usr/lib/system/libcompiler_rt.dylib
    0x7fff66511000 -     0x7fff6651affb  libcopyfile.dylib (146.50.5) <3885083D-50D8-3EEC-B481-B2E605180D7F> /usr/lib/system/libcopyfile.dylib
    0x7fff6651b000 -     0x7fff665a0ffb  libcorecrypto.dylib (562.70.3) <37AF8EC6-BF34-3D15-BC3A-A9B98F2427D4> /usr/lib/system/libcorecrypto.dylib
    0x7fff66628000 -     0x7fff66661ff7  libdispatch.dylib (913.60.3) <BF368549-2DFB-3530-B4CB-31D5EDAC4F2F> /usr/lib/system/libdispatch.dylib
    0x7fff66662000 -     0x7fff6667fff7  libdyld.dylib (551.5) <49BF9E96-8297-30CF-8AA6-128CC14054B2> /usr/lib/system/libdyld.dylib
    0x7fff66680000 -     0x7fff66680ffb  libkeymgr.dylib (28) <E34E283E-90FA-3C59-B48E-1277CDB9CDCE> /usr/lib/system/libkeymgr.dylib
    0x7fff66681000 -     0x7fff6668dff3  libkxld.dylib (4570.71.46) <0C2152D7-F14B-3DBC-B388-6A2530795DB0> /usr/lib/system/libkxld.dylib
    0x7fff6668e000 -     0x7fff6668eff7  liblaunch.dylib (1205.70.12) <4950A0B3-5EE1-3F9A-8B1A-94FB85FCCA2C> /usr/lib/system/liblaunch.dylib
    0x7fff6668f000 -     0x7fff66693ffb  libmacho.dylib (906) <1902A611-081A-3452-B11E-EBD1B166E831> /usr/lib/system/libmacho.dylib
    0x7fff66694000 -     0x7fff66696ff3  libquarantine.dylib (86) <26C0BA22-8F93-3A07-9A4E-C8D53D2CE42E> /usr/lib/system/libquarantine.dylib
    0x7fff66697000 -     0x7fff66698ff3  libremovefile.dylib (45) <711E18B2-5BBE-3211-A916-56740C27D17A> /usr/lib/system/libremovefile.dylib
    0x7fff66699000 -     0x7fff666b0fff  libsystem_asl.dylib (356.70.1) <39E46A6F-B228-3E78-B83E-1779F9707A39> /usr/lib/system/libsystem_asl.dylib
    0x7fff666b1000 -     0x7fff666b1fff  libsystem_blocks.dylib (67) <17303FDF-0D2D-3963-B05E-B4DF63052D47> /usr/lib/system/libsystem_blocks.dylib
    0x7fff666b2000 -     0x7fff6673bff7  libsystem_c.dylib (1244.50.9) <1187BFE8-4576-3247-8177-481554E1F9E7> /usr/lib/system/libsystem_c.dylib
    0x7fff6673c000 -     0x7fff6673fffb  libsystem_configuration.dylib (963.50.9) <A5DC1AFC-2C3E-3113-BE20-48F96064DB0F> /usr/lib/system/libsystem_configuration.dylib
    0x7fff66740000 -     0x7fff66743ffb  libsystem_coreservices.dylib (51) <486000D3-D8CB-3BE7-8EE5-8BF380DE6DF7> /usr/lib/system/libsystem_coreservices.dylib
    0x7fff66744000 -     0x7fff66745fff  libsystem_darwin.dylib (1244.50.9) <09C21A4A-9EE0-388B-A9D9-DFF8F6758791> /usr/lib/system/libsystem_darwin.dylib
    0x7fff66746000 -     0x7fff6674cff7  libsystem_dnssd.dylib (878.70.3) <E117689D-8CF3-3010-9EED-B71C033BF8D7> /usr/lib/system/libsystem_dnssd.dylib
    0x7fff6674d000 -     0x7fff66796ff7  libsystem_info.dylib (517.30.1) <AB634A98-B8AA-3804-8436-38261FC8EC4D> /usr/lib/system/libsystem_info.dylib
    0x7fff66797000 -     0x7fff667bdff7  libsystem_kernel.dylib (4570.71.46) <E71AB6DE-0D73-3C80-81D3-0268298C9681> /usr/lib/system/libsystem_kernel.dylib
    0x7fff667be000 -     0x7fff66809fcb  libsystem_m.dylib (3147.50.1) <17570F46-566C-39FC-BEF6-635A355DD549> /usr/lib/system/libsystem_m.dylib
    0x7fff6680a000 -     0x7fff66829ff7  libsystem_malloc.dylib (140.50.7) <A45A484C-3BDE-3BC4-A219-726E3CB332D9> /usr/lib/system/libsystem_malloc.dylib
    0x7fff6682a000 -     0x7fff6695aff7  libsystem_network.dylib (1229.70.2) <5E86B2DE-9E15-3354-8714-4094ED5F698D> /usr/lib/system/libsystem_network.dylib
    0x7fff6695b000 -     0x7fff66965ffb  libsystem_networkextension.dylib (767.70.2) <9DC03712-552D-3AEE-9519-B5ED70980B70> /usr/lib/system/libsystem_networkextension.dylib
    0x7fff66966000 -     0x7fff6696fff3  libsystem_notify.dylib (172) <08012EC0-2CD2-34BE-BF93-E7F56491299A> /usr/lib/system/libsystem_notify.dylib
    0x7fff66970000 -     0x7fff66977ff7  libsystem_platform.dylib (161.50.1) <6355EE2D-5456-3CA8-A227-B96E8F1E2AF8> /usr/lib/system/libsystem_platform.dylib
    0x7fff66978000 -     0x7fff66983fff  libsystem_pthread.dylib (301.50.1) <0E51CCBA-91F2-34E1-BF2A-FEEFD3D321E4> /usr/lib/system/libsystem_pthread.dylib
    0x7fff66984000 -     0x7fff66987fff  libsystem_sandbox.dylib (765.70.1) <553DFCC6-9D31-3B9C-AB7C-30F6F265786D> /usr/lib/system/libsystem_sandbox.dylib
    0x7fff66988000 -     0x7fff66989ff3  libsystem_secinit.dylib (30) <DE8D14E8-A276-3FF8-AE13-77F7040F33C1> /usr/lib/system/libsystem_secinit.dylib
    0x7fff6698a000 -     0x7fff66991ff7  libsystem_symptoms.dylib (820.60.3) <441C6CA0-5711-3BB1-8420-DDAC3D5272E1> /usr/lib/system/libsystem_symptoms.dylib
    0x7fff66992000 -     0x7fff669a5fff  libsystem_trace.dylib (829.70.1) <3A6CB706-8CA6-3616-8AFC-14AAD7FAF187> /usr/lib/system/libsystem_trace.dylib
    0x7fff669a7000 -     0x7fff669acff7  libunwind.dylib (35.3) <BEF3FB49-5604-3B5F-82B5-332B80023AC3> /usr/lib/system/libunwind.dylib
    0x7fff669ad000 -     0x7fff669daff7  libxpc.dylib (1205.70.12) <7B82608D-ED82-35C1-B13F-99855E25D6DC> /usr/lib/system/libxpc.dylib
Sample analysis of process 28931 written to file /dev/stdout
Sampling process 28853 for 3 seconds with 1 millisecond of run time between samples
Sampling completed, processing symbols...
Analysis of sampling zpool (pid 28853) every 1 millisecond
Process:         zpool [28853]
Path:            /usr/local/bin/zpool
Load Address:    0x10dd0d000
Identifier:      zpool
Version:         0
Code Type:       X86-64
Parent Process:  bash [7619]

Date/Time:       2019-10-23 11:17:33.786 +0200
Launch Time:     2019-10-23 10:43:44.515 +0200
OS Version:      Mac OS X 10.13.6 (17G8037)
Report Version:  7
Analysis Tool:   /usr/bin/sample

Physical footprint:         1204K
Physical footprint (peak):  1204K
----

Call graph:
    2745 Thread_3464907   DispatchQueue_1: com.apple.main-thread  (serial)
      2745 start  (in libdyld.dylib) + 1  [0x7fff66663015]
        2745 main  (in zpool) + 516  [0x10dd12b18]
          2745 zpool_do_remove  (in zpool) + 432  [0x10dd13f04]
            2745 zpool_vdev_remove  (in libzfs.2.dylib) + 336  [0x10e103068]
              2745 zfs_ioctl  (in libzfs.2.dylib) + 55  [0x10e110cd5]
                2745 ioctl  (in libsystem_kernel.dylib) + 151  [0x7fff667ad4eb]
                  2745 __ioctl  (in libsystem_kernel.dylib) + 10  [0x7fff667b33fa]

Total number in stack (recursive counted multiple, when >=5):

Sort by top of stack, same collapsed (when >= 5):
        __ioctl  (in libsystem_kernel.dylib)        2745

Binary Images:
       0x10dd0d000 -        0x10dd27fff +zpool (0) <D03B441A-11BB-35B4-A8EB-A9CBB5AF29F3> /usr/local/bin/zpool
       0x10dd3e000 -        0x10dd40fff +libdiskmgt.1.dylib (0) <C4DB976E-B3BD-3AC6-82C0-0BE510BF4D00> /usr/local/lib/libdiskmgt.1.dylib
       0x10dd4e000 -        0x10dd59fff +libnvpair.1.dylib (0) <7CA2EACE-A919-35AE-866E-16E595376714> /usr/local/lib/libnvpair.1.dylib
       0x10dd6b000 -        0x10dd7eff7 +libuutil.1.dylib (0) <38177B5B-F4C7-3F6E-AA7E-42CE44437EE8> /usr/local/lib/libuutil.1.dylib
       0x10dd98000 -        0x10df04fe3 +libzpool.1.dylib (0) <0B6180AA-AF77-3DD4-B9D8-B11F389BFD49> /usr/local/lib/libzpool.1.dylib
       0x10e0e7000 -        0x10e11fff7 +libzfs.2.dylib (0) <6CBC3221-8CE3-3431-91B4-027CA52574EA> /usr/local/lib/libzfs.2.dylib
       0x10e13e000 -        0x10e140fff +libzfs_core.1.dylib (0) <36F1BC6D-4A68-30C3-8374-358577C3A2BC> /usr/local/lib/libzfs_core.1.dylib
       0x1105d4000 -        0x11061eacf  dyld (551.5) <30B355CB-35BA-3112-AA76-4E46CD45F699> /usr/lib/dyld
    0x7fff3a8f0000 -     0x7fff3a8f0fff  com.apple.Accelerate (1.11 - Accelerate 1.11) <2EF00EFE-06E8-3082-85C3-2988B422A19D> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
    0x7fff3a909000 -     0x7fff3b057ffb  com.apple.vImage (8.1 - ???) <56C275C1-459F-37CD-BF29-2E6D81C29E53> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
    0x7fff3b058000 -     0x7fff3b1f8ff3  libBLAS.dylib (1211.50.2) <93259222-2D94-333D-A1EB-09C6FC4A64B4> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
    0x7fff3b1f9000 -     0x7fff3b233fef  libBNNS.dylib (38.1) <0AAE0109-71D5-3B26-8401-3E62DDF97624> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBNNS.dylib
    0x7fff3b234000 -     0x7fff3b60dff7  libLAPACK.dylib (1211.50.2) <70781D0B-AEAD-33EB-9900-10DC30F37D61> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
    0x7fff3b60e000 -     0x7fff3b624ff7  libLinearAlgebra.dylib (1211.50.2) <FB5A35EA-D636-3D69-AB53-B67E1EB8134C> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLinearAlgebra.dylib
    0x7fff3b625000 -     0x7fff3b62aff3  libQuadrature.dylib (3) <3D6BF66A-55B2-3692-BAC7-DEB0C676ED29> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libQuadrature.dylib
    0x7fff3b62b000 -     0x7fff3b6abfff  libSparse.dylib (79.50.2) <0DC25CDD-F8C1-3D6E-B472-8B060708424F> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparse.dylib
    0x7fff3b6ac000 -     0x7fff3b6beff7  libSparseBLAS.dylib (1211.50.2) <994A7D7E-FA3B-3943-A59C-8150B9AF86BE> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libSparseBLAS.dylib
    0x7fff3b6bf000 -     0x7fff3b832fe7  libvDSP.dylib (622.50.5) <4BF310F4-31A3-3DA5-80E4-7F8014AD380B> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
    0x7fff3b833000 -     0x7fff3b8e9fef  libvMisc.dylib (622.50.5) <88143A88-4FDE-35A1-85FE-54FF0C2A9E43> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
    0x7fff3b8ea000 -     0x7fff3b8eafff  com.apple.Accelerate.vecLib (3.11 - vecLib 3.11) <880DEF40-389A-3F56-ACA1-B62A5BC57E59> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
    0x7fff3ca8e000 -     0x7fff3ca8efff  com.apple.ApplicationServices (48 - 50) <63B0C61F-CFE8-3884-853F-E6FB6CD84A44> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/ApplicationServices
    0x7fff3ca8f000 -     0x7fff3caf5fff  com.apple.ApplicationServices.ATS (377 - 445.5) <703CE7E4-426A-35C0-A229-F140F30F5340> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/ATS
    0x7fff3cb8e000 -     0x7fff3ccb0fff  libFontParser.dylib (222.1.7) <9C4CEFB4-0F08-3B60-BD50-0120A37F6EA2> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontParser.dylib
    0x7fff3ccb1000 -     0x7fff3ccfbff7  libFontRegistry.dylib (221.5) <8F68EA59-C8EE-3FA3-BD19-0F1A58441440> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ATS.framework/Versions/A/Resources/libFontRegistry.dylib
    0x7fff3ce3d000 -     0x7fff3ce41ff3  com.apple.ColorSyncLegacy (4.13.0 - 1) <A5FB2694-1559-34A8-A3D3-2029F68A63CA> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/ColorSyncLegacy.framework/Versions/A/ColorSyncLegacy
    0x7fff3cee1000 -     0x7fff3cf33ffb  com.apple.HIServices (1.22 - 625) <78A1FB4F-A39C-3159-B285-600C997335FB> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/HIServices.framework/Versions/A/HIServices
    0x7fff3cf34000 -     0x7fff3cf42fff  com.apple.LangAnalysis (1.7.0 - 1.7.0) <B65FF7E6-E9B5-34D8-8CA7-63D415A8A9A6> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/LangAnalysis.framework/Versions/A/LangAnalysis
    0x7fff3cf43000 -     0x7fff3cf8ffff  com.apple.print.framework.PrintCore (13.4 - 503.2) <B90C67C1-0292-3CEC-885D-F1882CD104BE> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/PrintCore.framework/Versions/A/PrintCore
    0x7fff3cf90000 -     0x7fff3cfcafff  com.apple.QD (3.12 - 404.2) <38B20AFF-9D54-3B52-A6DC-C0D71380AA5F> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/QD.framework/Versions/A/QD
    0x7fff3cfcb000 -     0x7fff3cfd7fff  com.apple.speech.synthesis.framework (7.8.1 - 7.8.1) <A08DE016-C8F2-3B0E-BD34-15959D13DBF0> /System/Library/Frameworks/ApplicationServices.framework/Versions/A/Frameworks/SpeechSynthesis.framework/Versions/A/SpeechSynthesis
    0x7fff3cfd8000 -     0x7fff3d266ff7  com.apple.audio.toolbox.AudioToolbox (1.14 - 1.14) <E0B8B5D8-80A0-308B-ABD6-F8612102B5D8> /System/Library/Frameworks/AudioToolbox.framework/Versions/A/AudioToolbox
    0x7fff3d58b000 -     0x7fff3d926ff7  com.apple.CFNetwork (902.3.1 - 902.3.1) <4C012538-BB8A-32F3-AACA-011092BEFAB1> /System/Library/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
    0x7fff3de5b000 -     0x7fff3df14fff  com.apple.ColorSync (4.13.0 - 3325) <D283C285-447D-3258-A7E4-59532123B8FF> /System/Library/Frameworks/ColorSync.framework/Versions/A/ColorSync
    0x7fff3e0a1000 -     0x7fff3e134ff7  com.apple.audio.CoreAudio (4.3.0 - 4.3.0) <EB35D3EC-56EA-33E6-98DC-BDC3A5FA8ACE> /System/Library/Frameworks/CoreAudio.framework/Versions/A/CoreAudio
    0x7fff3e1c5000 -     0x7fff3e529ff3  com.apple.CoreData (120 - 851) <8D605466-8676-3D45-9A4B-8980971551C3> /System/Library/Frameworks/CoreData.framework/Versions/A/CoreData
    0x7fff3e52a000 -     0x7fff3e60dfff  com.apple.CoreDisplay (99.14 - 99.14) <A1B91ADD-828D-33A0-8A92-CC3F83DF89D0> /System/Library/Frameworks/CoreDisplay.framework/Versions/A/CoreDisplay
    0x7fff3e60e000 -     0x7fff3eaaafff  com.apple.CoreFoundation (6.9 - 1455.11) <EA9F0BF7-6A1F-36C9-A42E-6DE3DBCD99BD> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
    0x7fff3eaac000 -     0x7fff3f0daff7  com.apple.CoreGraphics (2.0 - 1161.21.2) <703FBD9C-F88A-3820-A322-49B9C765715D> /System/Library/Frameworks/CoreGraphics.framework/Versions/A/CoreGraphics
    0x7fff3f0dc000 -     0x7fff3f3cbfff  com.apple.CoreImage (13.0.0 - 579.5) <AAE2DFD0-9B0A-3D56-8A3E-C460BAF70394> /System/Library/Frameworks/CoreImage.framework/Versions/A/CoreImage
    0x7fff3f7a0000 -     0x7fff3f7a0fff  com.apple.CoreServices (822.37 - 822.37) <C7BB281A-16A2-321A-90F4-B970278D9C2A> /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
    0x7fff3f7a1000 -     0x7fff3f815ffb  com.apple.AE (735.2 - 735.2) <690A6F47-1306-371F-B257-783D8F83EA9E> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
    0x7fff3f816000 -     0x7fff3faedfff  com.apple.CoreServices.CarbonCore (1178.4.2 - 1178.4.2) <08AF7B7B-219B-3957-86CE-C96B423C823C> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
    0x7fff3faee000 -     0x7fff3fb22fff  com.apple.DictionaryServices (1.2 - 284.2) <6505B075-41C3-3C62-A4C3-85CE3F6825CD> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
    0x7fff3fb23000 -     0x7fff3fb2bffb  com.apple.CoreServices.FSEvents (1239.50.1 - 1239.50.1) <3637CEC7-DF0E-320E-9634-44A442925C65> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/FSEvents.framework/Versions/A/FSEvents
    0x7fff3fb2c000 -     0x7fff3fceaff7  com.apple.LaunchServices (822.37 - 822.37) <6AA93307-220A-3417-BBC2-FE1C0DE0B652> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
    0x7fff3fceb000 -     0x7fff3fd9bfff  com.apple.Metadata (10.7.0 - 1191.7) <3DD530A7-E104-3469-98BB-20449834B2FE> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
    0x7fff3fd9c000 -     0x7fff3fdfcfff  com.apple.CoreServices.OSServices (822.37 - 822.37) <4AD2FC98-C6CB-392A-A22F-196A723D7FAE> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
    0x7fff3fdfd000 -     0x7fff3fe6bfff  com.apple.SearchKit (1.4.0 - 1.4.0) <3662545A-B1CF-3079-BDCD-C83855CEFEEE> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
    0x7fff3fe6c000 -     0x7fff3fe90ffb  com.apple.coreservices.SharedFileList (71.21 - 71.21) <1B5228EF-D869-3A50-A373-7F4B0289FADD> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SharedFileList.framework/Versions/A/SharedFileList
    0x7fff40131000 -     0x7fff40281fff  com.apple.CoreText (352.0 - 578.22) <6129F39D-284D-3BBF-8999-7854AB61C01C> /System/Library/Frameworks/CoreText.framework/Versions/A/CoreText
    0x7fff40282000 -     0x7fff402bcfff  com.apple.CoreVideo (1.8 - 0.0) <86CCC036-51BB-3DD1-9601-D93798BCCD0F> /System/Library/Frameworks/CoreVideo.framework/Versions/A/CoreVideo
    0x7fff405c3000 -     0x7fff405c8fff  com.apple.DiskArbitration (2.7 - 2.7) <2F628584-8177-3894-92CE-9CA6089B61C6> /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
    0x7fff40789000 -     0x7fff40b50ffb  com.apple.Foundation (6.9 - 1455.11) <121FC177-8542-36BA-9F72-CCEAE8AF88FE> /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
    0x7fff40bc1000 -     0x7fff40bf1ff3  com.apple.GSS (4.0 - 2.0) <FC69A120-3C19-3802-87B4-C29217758143> /System/Library/Frameworks/GSS.framework/Versions/A/GSS
    0x7fff40e67000 -     0x7fff40f02fff  com.apple.framework.IOKit (2.0.2 - 1445.71.6) <C68CE7ED-85CA-3D1F-BAA0-908327BEC0DE> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
    0x7fff40f04000 -     0x7fff40f0bfff  com.apple.IOSurface (211.15 - 211.15) <9FD406F1-6BF2-35B0-8339-DF83A1A661EB> /System/Library/Frameworks/IOSurface.framework/Versions/A/IOSurface
    0x7fff40f62000 -     0x7fff410dffff  com.apple.ImageIO.framework (3.3.0 - 1739.3.1) <3564ADD2-803F-35CC-AB7F-6F41241010FF> /System/Library/Frameworks/ImageIO.framework/Versions/A/ImageIO
    0x7fff410e0000 -     0x7fff410e4ffb  libGIF.dylib (1739.3.1) <3CB9E6D5-FF76-37E5-921D-57F2C36D8CFC> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libGIF.dylib
    0x7fff410e5000 -     0x7fff411ccff7  libJP2.dylib (1739.3.1) <2B233EA0-59CC-3D7D-81AF-F471448F5271> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJP2.dylib
    0x7fff411cd000 -     0x7fff411f0fff  libJPEG.dylib (1739.3.1) <4B3C1085-9177-3872-BCAF-965CBCD6A906> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libJPEG.dylib
    0x7fff414cd000 -     0x7fff414f3ff3  libPng.dylib (1739.3.1) <734D972E-0BD6-3DAC-A465-43A8AF9E0BC8> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libPng.dylib
    0x7fff414f4000 -     0x7fff414f6ff3  libRadiance.dylib (1739.3.1) <99967613-358F-377D-BEC2-C52CC2B18811> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libRadiance.dylib
    0x7fff414f7000 -     0x7fff41544ff3  libTIFF.dylib (1739.3.1) <BEAC3497-7D7F-3A09-B232-07E9792ACF88> /System/Library/Frameworks/ImageIO.framework/Versions/A/Resources/libTIFF.dylib
    0x7fff423fe000 -     0x7fff42417ff7  com.apple.Kerberos (3.0 - 1) <F86DCCDF-93C1-38B3-82C2-477C12E8EE6D> /System/Library/Frameworks/Kerberos.framework/Versions/A/Kerberos
    0x7fff42df9000 -     0x7fff42e7afff  com.apple.Metal (125.30 - 125.30) <6B9EBDEE-C64C-3C1C-922A-0363B642C9BC> /System/Library/Frameworks/Metal.framework/Versions/A/Metal
    0x7fff42e97000 -     0x7fff42eb2fff  com.apple.MetalPerformanceShaders.MPSCore (1.0 - 1) <AD754E8F-CA00-3878-9AF3-208C224A230B> /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSCore.framework/Versions/A/MPSCore
    0x7fff42eb3000 -     0x7fff42f22fef  com.apple.MetalPerformanceShaders.MPSImage (1.0 - 1) <338B7779-E608-3D68-8A07-2ACC11299744> /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSImage.framework/Versions/A/MPSImage
    0x7fff42f23000 -     0x7fff42f47fff  com.apple.MetalPerformanceShaders.MPSMatrix (1.0 - 1) <9CE072D7-853B-3939-9645-7EB951376B87> /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSMatrix.framework/Versions/A/MPSMatrix
    0x7fff42f48000 -     0x7fff4302fff7  com.apple.MetalPerformanceShaders.MPSNeuralNetwork (1.0 - 1) <0DE891AD-27E5-38FF-AEC8-4A95356C4357> /System/Library/Frameworks/MetalPerformanceShaders.framework/Frameworks/MPSNeuralNetwork.framework/Versions/A/MPSNeuralNetwork
    0x7fff43030000 -     0x7fff43030ff7  com.apple.MetalPerformanceShaders.MetalPerformanceShaders (1.0 - 1) <2D2D261C-50B0-32F9-BF9A-5C01382BB528> /System/Library/Frameworks/MetalPerformanceShaders.framework/Versions/A/MetalPerformanceShaders
    0x7fff4402f000 -     0x7fff4403bffb  com.apple.NetFS (6.0 - 4.0) <471DD96F-FA2E-3FE9-9746-2519A6780D1A> /System/Library/Frameworks/NetFS.framework/Versions/A/NetFS
    0x7fff46e2d000 -     0x7fff46e87ff7  com.apple.opencl (2.8.24 - 2.8.24) <CF922AFD-CBA4-3B23-A0C1-E3E5362BB6C7> /System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL
    0x7fff46e88000 -     0x7fff46ea4ffb  com.apple.CFOpenDirectory (10.13 - 207.50.1) <29F55F7B-379F-3053-8FF3-5C6675A3DD4D> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/Frameworks/CFOpenDirectory.framework/Versions/A/CFOpenDirectory
    0x7fff46ea5000 -     0x7fff46eb0fff  com.apple.OpenDirectory (10.13 - 207.50.1) <F895547D-4915-353F-9C1E-E95172BA803B> /System/Library/Frameworks/OpenDirectory.framework/Versions/A/OpenDirectory
    0x7fff4802f000 -     0x7fff48031fff  libCVMSPluginSupport.dylib (16.7.4) <6846D939-478C-3AE3-9DCD-CD4F4CD05409> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCVMSPluginSupport.dylib
    0x7fff48032000 -     0x7fff48037ffb  libCoreFSCache.dylib (162.9) <DD9DD721-2957-3D05-B361-70AF9EBFB280> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreFSCache.dylib
    0x7fff48038000 -     0x7fff4803cfff  libCoreVMClient.dylib (162.9) <4E4F3EA2-5B53-31A3-8256-54EFAA94E4D6> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libCoreVMClient.dylib
    0x7fff4803d000 -     0x7fff48046ff3  libGFXShared.dylib (16.7.4) <6187AD8B-1CD2-3775-84CD-76A0EA8BCDD4> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGFXShared.dylib
    0x7fff48047000 -     0x7fff48052fff  libGL.dylib (16.7.4) <2BB333D3-5C61-33DF-8545-06DF2D08B83D> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGL.dylib
    0x7fff48053000 -     0x7fff4808efe7  libGLImage.dylib (16.7.4) <4DA003CE-0B74-3FE4-808C-B2FBCE517EB4> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLImage.dylib
    0x7fff481fd000 -     0x7fff4823bffb  libGLU.dylib (16.7.4) <BCB09CD8-EB0E-38FA-8B5A-9E29532EE364> /System/Library/Frameworks/OpenGL.framework/Versions/A/Libraries/libGLU.dylib
    0x7fff48bb3000 -     0x7fff48bc2ff3  com.apple.opengl (16.7.4 - 16.7.4) <7B71A596-B6D8-335A-B7CA-8684D83B9C0E> /System/Library/Frameworks/OpenGL.framework/Versions/A/OpenGL
    0x7fff49a11000 -     0x7fff49c5dff7  com.apple.QuartzCore (1.11 - 584.64.2) <DDA39332-DE09-3737-A332-0002AE05DD39> /System/Library/Frameworks/QuartzCore.framework/Versions/A/QuartzCore
    0x7fff4a495000 -     0x7fff4a7c0fff  com.apple.security (7.0 - 58286.70.14) <CAAA4591-34B7-3281-97B9-0D4BCFBB7816> /System/Library/Frameworks/Security.framework/Versions/A/Security
    0x7fff4a7c1000 -     0x7fff4a84dff7  com.apple.securityfoundation (6.0 - 55185.50.5) <B5F933AE-BF0B-36CB-B2D4-3C07DD952BCE> /System/Library/Frameworks/SecurityFoundation.framework/Versions/A/SecurityFoundation
    0x7fff4a87f000 -     0x7fff4a883ffb  com.apple.xpc.ServiceManagement (1.0 - 1) <E3146BBE-7F32-3605-AD2B-54C0E70759DC> /System/Library/Frameworks/ServiceManagement.framework/Versions/A/ServiceManagement
    0x7fff4ac28000 -     0x7fff4ac98ff3  com.apple.SystemConfiguration (1.17 - 1.17) <CD4A4C66-7886-3323-A3B9-0C8952500856> /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
    0x7fff4db95000 -     0x7fff4dc28fff  com.apple.APFS (1.0 - 1) <7CC77BF1-881B-357A-9E05-A73C661F830E> /System/Library/PrivateFrameworks/APFS.framework/Versions/A/APFS
    0x7fff4e9d2000 -     0x7fff4ea1aff3  com.apple.AppleJPEG (1.0 - 1) <8DD410CB-76A1-3F22-9A9F-0491FA0CEB4A> /System/Library/PrivateFrameworks/AppleJPEG.framework/Versions/A/AppleJPEG
    0x7fff4ea55000 -     0x7fff4ea7dfff  com.apple.applesauce (1.0 - ???) <CCA8B094-1BCE-3AE3-A0A7-D544C818DE36> /System/Library/PrivateFrameworks/AppleSauce.framework/Versions/A/AppleSauce
    0x7fff4f2a0000 -     0x7fff4f2a7ff7  com.apple.coreservices.BackgroundTaskManagement (1.0 - 57.1) <51A41CA3-DB1D-3380-993E-99C54AEE518E> /System/Library/PrivateFrameworks/BackgroundTaskManagement.framework/Versions/A/BackgroundTaskManagement
    0x7fff50ce9000 -     0x7fff50cf2ff3  com.apple.CommonAuth (4.0 - 2.0) <21AF4AC2-E650-35F9-AB33-3EBA2769FBC0> /System/Library/PrivateFrameworks/CommonAuth.framework/Versions/A/CommonAuth
    0x7fff51806000 -     0x7fff51816ff7  com.apple.CoreEmoji (1.0 - 69.3) <A4357F5C-0C38-3A61-B456-D7321EB2CEE5> /System/Library/PrivateFrameworks/CoreEmoji.framework/Versions/A/CoreEmoji
    0x7fff529be000 -     0x7fff529c2ffb  com.apple.DSExternalDisplay (3.1 - 380) <901B7F6D-376A-3848-99D0-170C4D00F776> /System/Library/PrivateFrameworks/DSExternalDisplay.framework/Versions/A/DSExternalDisplay
    0x7fff53a8d000 -     0x7fff53ebbfff  com.apple.vision.FaceCore (3.3.2 - 3.3.2) <B574FE33-4A41-3611-9738-388EBAF03E37> /System/Library/PrivateFrameworks/FaceCore.framework/Versions/A/FaceCore
    0x7fff57183000 -     0x7fff57188fff  com.apple.GPUWrangler (3.20.16 - 3.20.16) <C24B2DA4-998E-3880-9C73-EB93AC04C357> /System/Library/PrivateFrameworks/GPUWrangler.framework/Versions/A/GPUWrangler
    0x7fff57efe000 -     0x7fff57f0dfff  com.apple.GraphVisualizer (1.0 - 5) <B993B8A2-5700-3DFC-9EB7-4CCEE8F959F1> /System/Library/PrivateFrameworks/GraphVisualizer.framework/Versions/A/GraphVisualizer
    0x7fff57f90000 -     0x7fff58004fff  com.apple.Heimdal (4.0 - 2.0) <BAE3D978-2285-35DF-907E-02BC37986D02> /System/Library/PrivateFrameworks/Heimdal.framework/Versions/A/Heimdal
    0x7fff58917000 -     0x7fff5891eff7  com.apple.IOAccelerator (378.28 - 378.28) <BE5CBEAA-314E-348B-BE85-7E4B2E4C42FF> /System/Library/PrivateFrameworks/IOAccelerator.framework/Versions/A/IOAccelerator
    0x7fff58922000 -     0x7fff58939fff  com.apple.IOPresentment (1.0 - 35.1) <53B47F2A-9A9A-33F5-ADCB-FF766292B93A> /System/Library/PrivateFrameworks/IOPresentment.framework/Versions/A/IOPresentment
    0x7fff58fc3000 -     0x7fff590b8ff7  com.apple.LanguageModeling (1.0 - 159.5.3) <7F0AC200-E3DD-39FB-8A95-00DD70B66A9F> /System/Library/PrivateFrameworks/LanguageModeling.framework/Versions/A/LanguageModeling
    0x7fff590b9000 -     0x7fff590fbfff  com.apple.Lexicon-framework (1.0 - 33.5) <DC94CF9E-1EB4-3C0E-B298-CA1190885276> /System/Library/PrivateFrameworks/Lexicon.framework/Versions/A/Lexicon
    0x7fff590ff000 -     0x7fff59106ff7  com.apple.LinguisticData (1.0 - 238.3) <49A54649-1021-3DBD-99B8-1B2EDFFA5378> /System/Library/PrivateFrameworks/LinguisticData.framework/Versions/A/LinguisticData
    0x7fff59e19000 -     0x7fff59e82ff7  com.apple.gpusw.MetalTools (1.0 - 1) <B4217182-B2AA-3AA3-80E8-F4C80B71BDDC> /System/Library/PrivateFrameworks/MetalTools.framework/Versions/A/MetalTools
    0x7fff5a0b7000 -     0x7fff5a0e1ffb  com.apple.MultitouchSupport.framework (1614.1 - 1614.1) <A0709B43-FA9E-3617-8E7A-D68CDFAB2167> /System/Library/PrivateFrameworks/MultitouchSupport.framework/Versions/A/MultitouchSupport
    0x7fff5a348000 -     0x7fff5a353fff  com.apple.NetAuth (6.2 - 6.2) <2CFA1228-61A1-3708-988D-30499A1D6A9F> /System/Library/PrivateFrameworks/NetAuth.framework/Versions/A/NetAuth
    0x7fff6039b000 -     0x7fff60637fff  com.apple.SkyLight (1.600.0 - 312.103.11) <2E04D36B-F539-356A-BED5-781CD98C2C96> /System/Library/PrivateFrameworks/SkyLight.framework/Versions/A/SkyLight
    0x7fff61fac000 -     0x7fff61fb4ff7  com.apple.TCC (1.0 - 1) <E1EB7272-FE6F-39AB-83CA-B2B5F2A88D9B> /System/Library/PrivateFrameworks/TCC.framework/Versions/A/TCC
    0x7fff63c5b000 -     0x7fff63c5dffb  com.apple.loginsupport (1.0 - 1) <D1232C1B-80EA-3DF8-9466-013695D0846E> /System/Library/PrivateFrameworks/login.framework/Versions/A/Frameworks/loginsupport.framework/Versions/A/loginsupport
    0x7fff63dc4000 -     0x7fff63df7ff7  libclosured.dylib (551.5) <112BC241-6626-3848-8DD8-B34B5B6F7ABC> /usr/lib/closure/libclosured.dylib
    0x7fff63eb1000 -     0x7fff63eeaff7  libCRFSuite.dylib (41) <FE5EDB68-2593-3C2E-BBAF-1C52D206F296> /usr/lib/libCRFSuite.dylib
    0x7fff63eeb000 -     0x7fff63ef6fff  libChineseTokenizer.dylib (28) <53633C9B-A3A8-36F7-A53C-432D802F4BB8> /usr/lib/libChineseTokenizer.dylib
    0x7fff63f88000 -     0x7fff63f89ff3  libDiagnosticMessagesClient.dylib (104) <9712E980-76EE-3A89-AEA6-DF4BAF5C0574> /usr/lib/libDiagnosticMessagesClient.dylib
    0x7fff63fc0000 -     0x7fff6418aff3  libFosl_dynamic.dylib (17.8) <C58ED77A-4986-31C2-994C-34DDFB8106F0> /usr/lib/libFosl_dynamic.dylib
    0x7fff641c2000 -     0x7fff641c2fff  libOpenScriptingUtil.dylib (174) <610F0242-7CE5-3C86-951B-B646562694AF> /usr/lib/libOpenScriptingUtil.dylib
    0x7fff642fe000 -     0x7fff642ffffb  libSystem.B.dylib (1252.50.4) <C084DB19-08B8-322B-9344-67F2970929DA> /usr/lib/libSystem.B.dylib
    0x7fff64392000 -     0x7fff64392fff  libapple_crypto.dylib (109.50.14) <48BA2E76-BF2F-3522-A54E-D7FB7EAF7A57> /usr/lib/libapple_crypto.dylib
    0x7fff64393000 -     0x7fff643a9ff7  libapple_nghttp2.dylib (1.24) <01402BC4-4822-3676-9C80-50D83F816424> /usr/lib/libapple_nghttp2.dylib
    0x7fff643aa000 -     0x7fff643d4ff3  libarchive.2.dylib (54) <8FC28DD8-E315-3C3E-95FE-D1D2CBE49888> /usr/lib/libarchive.2.dylib
    0x7fff644dc000 -     0x7fff644dcff3  libauto.dylib (187) <A05C7900-F8C7-3E75-8D3F-909B40C19717> /usr/lib/libauto.dylib
    0x7fff644dd000 -     0x7fff64595ff3  libboringssl.dylib (109.50.14) <E6813F87-B5E4-3F7F-A725-E6A7F2BD02EC> /usr/lib/libboringssl.dylib
    0x7fff64596000 -     0x7fff645a6ff3  libbsm.0.dylib (39) <6BC96A72-AFBE-34FD-91B1-748A530D8AE6> /usr/lib/libbsm.0.dylib
    0x7fff645a7000 -     0x7fff645b4ffb  libbz2.1.0.dylib (38) <0A5086BB-4724-3C14-979D-5AD4F26B5B45> /usr/lib/libbz2.1.0.dylib
    0x7fff645b5000 -     0x7fff6460bfff  libc++.1.dylib (400.9) <7D3DACCC-3804-393C-ABC1-1A580FD00CB6> /usr/lib/libc++.1.dylib
    0x7fff6460c000 -     0x7fff64630ff7  libc++abi.dylib (400.8.2) <EF5E37D7-11D9-3530-BE45-B986612D13E2> /usr/lib/libc++abi.dylib
    0x7fff64632000 -     0x7fff64642fff  libcmph.dylib (6) <890DEC4C-4334-393C-8B56-7C8560BBED9D> /usr/lib/libcmph.dylib
    0x7fff64643000 -     0x7fff6465afd3  libcompression.dylib (47.60.2) <CC7ACB0A-49CD-3564-978F-49F843186F27> /usr/lib/libcompression.dylib
    0x7fff64905000 -     0x7fff6491dff7  libcoretls.dylib (155.50.1) <D350052E-DC4D-3185-ADBA-BA48EDCEE955> /usr/lib/libcoretls.dylib
    0x7fff6491e000 -     0x7fff6491fff3  libcoretls_cfhelpers.dylib (155.50.1) <B297F5D8-F2FE-3566-A752-E9D998B9C039> /usr/lib/libcoretls_cfhelpers.dylib
    0x7fff64df0000 -     0x7fff64e45ff3  libcups.2.dylib (462.2.7) <46179788-4363-38B2-9204-E8AC8123D899> /usr/lib/libcups.2.dylib
    0x7fff64f85000 -     0x7fff64f85fff  libenergytrace.dylib (16) <A92AB8B8-B986-3CE6-980D-D55090FEF387> /usr/lib/libenergytrace.dylib
    0x7fff64fbc000 -     0x7fff64fc1ff3  libheimdal-asn1.dylib (520.50.9) <EF65C17E-31CD-31DC-876C-BF3EF393099C> /usr/lib/libheimdal-asn1.dylib
    0x7fff64fed000 -     0x7fff650deff7  libiconv.2.dylib (51.50.1) <2FEC9707-3FAF-3828-A50D-8605086D060F> /usr/lib/libiconv.2.dylib
    0x7fff650df000 -     0x7fff65306ffb  libicucore.A.dylib (59181.0.1) <2CE6205F-D375-3BF5-AA0A-3254BC4773D0> /usr/lib/libicucore.A.dylib
    0x7fff65353000 -     0x7fff65354fff  liblangid.dylib (128) <39C39393-0D05-301D-93B2-F224FC4949AA> /usr/lib/liblangid.dylib
    0x7fff65355000 -     0x7fff6536effb  liblzma.5.dylib (10) <3D419A50-961F-37D2-8A01-3DC7AB7B8D18> /usr/lib/liblzma.5.dylib
    0x7fff6536f000 -     0x7fff65385ff7  libmarisa.dylib (9) <D6D2D55D-1D2E-3442-B152-B18803C0ABB4> /usr/lib/libmarisa.dylib
    0x7fff65436000 -     0x7fff6565eff7  libmecabra.dylib (779.7.6) <F462F170-E872-3D09-B219-973D5E99C09F> /usr/lib/libmecabra.dylib
    0x7fff65836000 -     0x7fff659b1fff  libnetwork.dylib (1229.70.2) <E185D902-AC7F-3044-87C0-AE2887C59CE7> /usr/lib/libnetwork.dylib
    0x7fff65a38000 -     0x7fff65e293b7  libobjc.A.dylib (723) <BEEA51E2-1C4D-3672-87F6-F267B4AC6221> /usr/lib/libobjc.A.dylib
    0x7fff65e3c000 -     0x7fff65e40fff  libpam.2.dylib (22) <7B4D2CE2-1438-387A-9802-5CEEFBF26F86> /usr/lib/libpam.2.dylib
    0x7fff65e43000 -     0x7fff65e77fff  libpcap.A.dylib (79.20.1) <FA13918B-A247-3181-B256-9B852C7BA316> /usr/lib/libpcap.A.dylib
    0x7fff65ef6000 -     0x7fff65f12ffb  libresolv.9.dylib (65) <E8F3415B-4472-3202-8901-41FD87981DB2> /usr/lib/libresolv.9.dylib
    0x7fff65f63000 -     0x7fff660f0ff7  libsqlite3.dylib (274.8.1) <B70A2908-E0F7-364E-952A-1D2D19787B89> /usr/lib/libsqlite3.dylib
    0x7fff662c4000 -     0x7fff66324ff3  libusrtcp.dylib (1229.70.2) <1E065228-D0E3-3808-9405-894056C6BEC0> /usr/lib/libusrtcp.dylib
    0x7fff66325000 -     0x7fff66328ffb  libutil.dylib (51.20.1) <216D18E5-0BAF-3EAF-A38E-F6AC37CBABD9> /usr/lib/libutil.dylib
    0x7fff66329000 -     0x7fff66336ff3  libxar.1.dylib (417.1) <351781D5-3CCF-320F-BB5A-4DFCC1CCD31D> /usr/lib/libxar.1.dylib
    0x7fff6633a000 -     0x7fff66421fff  libxml2.2.dylib (31.14) <4B6055E6-D8AD-342F-BCD7-F2EDD9069014> /usr/lib/libxml2.2.dylib
    0x7fff66422000 -     0x7fff6644afff  libxslt.1.dylib (15.14) <AECAB47A-B160-3D6D-BDDC-24B05C6EB8D4> /usr/lib/libxslt.1.dylib
    0x7fff6644b000 -     0x7fff6645dffb  libz.1.dylib (70) <48C67CFC-940D-3857-8DAD-857774605352> /usr/lib/libz.1.dylib
    0x7fff664f9000 -     0x7fff664fdff7  libcache.dylib (80) <092479CB-1008-3A83-BECF-E115F24D13C1> /usr/lib/system/libcache.dylib
    0x7fff664fe000 -     0x7fff66508ff3  libcommonCrypto.dylib (60118.50.1) <029F5985-9B6E-3DCB-9B96-FD007678C6A7> /usr/lib/system/libcommonCrypto.dylib
    0x7fff66509000 -     0x7fff66510fff  libcompiler_rt.dylib (62) <968B8E3F-3681-3230-9D78-BB8732024F6E> /usr/lib/system/libcompiler_rt.dylib
    0x7fff66511000 -     0x7fff6651affb  libcopyfile.dylib (146.50.5) <3885083D-50D8-3EEC-B481-B2E605180D7F> /usr/lib/system/libcopyfile.dylib
    0x7fff6651b000 -     0x7fff665a0ffb  libcorecrypto.dylib (562.70.3) <37AF8EC6-BF34-3D15-BC3A-A9B98F2427D4> /usr/lib/system/libcorecrypto.dylib
    0x7fff66628000 -     0x7fff66661ff7  libdispatch.dylib (913.60.3) <BF368549-2DFB-3530-B4CB-31D5EDAC4F2F> /usr/lib/system/libdispatch.dylib
    0x7fff66662000 -     0x7fff6667fff7  libdyld.dylib (551.5) <49BF9E96-8297-30CF-8AA6-128CC14054B2> /usr/lib/system/libdyld.dylib
    0x7fff66680000 -     0x7fff66680ffb  libkeymgr.dylib (28) <E34E283E-90FA-3C59-B48E-1277CDB9CDCE> /usr/lib/system/libkeymgr.dylib
    0x7fff66681000 -     0x7fff6668dff3  libkxld.dylib (4570.71.46) <0C2152D7-F14B-3DBC-B388-6A2530795DB0> /usr/lib/system/libkxld.dylib
    0x7fff6668e000 -     0x7fff6668eff7  liblaunch.dylib (1205.70.12) <4950A0B3-5EE1-3F9A-8B1A-94FB85FCCA2C> /usr/lib/system/liblaunch.dylib
    0x7fff6668f000 -     0x7fff66693ffb  libmacho.dylib (906) <1902A611-081A-3452-B11E-EBD1B166E831> /usr/lib/system/libmacho.dylib
    0x7fff66694000 -     0x7fff66696ff3  libquarantine.dylib (86) <26C0BA22-8F93-3A07-9A4E-C8D53D2CE42E> /usr/lib/system/libquarantine.dylib
    0x7fff66697000 -     0x7fff66698ff3  libremovefile.dylib (45) <711E18B2-5BBE-3211-A916-56740C27D17A> /usr/lib/system/libremovefile.dylib
    0x7fff66699000 -     0x7fff666b0fff  libsystem_asl.dylib (356.70.1) <39E46A6F-B228-3E78-B83E-1779F9707A39> /usr/lib/system/libsystem_asl.dylib
    0x7fff666b1000 -     0x7fff666b1fff  libsystem_blocks.dylib (67) <17303FDF-0D2D-3963-B05E-B4DF63052D47> /usr/lib/system/libsystem_blocks.dylib
    0x7fff666b2000 -     0x7fff6673bff7  libsystem_c.dylib (1244.50.9) <1187BFE8-4576-3247-8177-481554E1F9E7> /usr/lib/system/libsystem_c.dylib
    0x7fff6673c000 -     0x7fff6673fffb  libsystem_configuration.dylib (963.50.9) <A5DC1AFC-2C3E-3113-BE20-48F96064DB0F> /usr/lib/system/libsystem_configuration.dylib
    0x7fff66740000 -     0x7fff66743ffb  libsystem_coreservices.dylib (51) <486000D3-D8CB-3BE7-8EE5-8BF380DE6DF7> /usr/lib/system/libsystem_coreservices.dylib
    0x7fff66744000 -     0x7fff66745fff  libsystem_darwin.dylib (1244.50.9) <09C21A4A-9EE0-388B-A9D9-DFF8F6758791> /usr/lib/system/libsystem_darwin.dylib
    0x7fff66746000 -     0x7fff6674cff7  libsystem_dnssd.dylib (878.70.3) <E117689D-8CF3-3010-9EED-B71C033BF8D7> /usr/lib/system/libsystem_dnssd.dylib
    0x7fff6674d000 -     0x7fff66796ff7  libsystem_info.dylib (517.30.1) <AB634A98-B8AA-3804-8436-38261FC8EC4D> /usr/lib/system/libsystem_info.dylib
    0x7fff66797000 -     0x7fff667bdff7  libsystem_kernel.dylib (4570.71.46) <E71AB6DE-0D73-3C80-81D3-0268298C9681> /usr/lib/system/libsystem_kernel.dylib
    0x7fff667be000 -     0x7fff66809fcb  libsystem_m.dylib (3147.50.1) <17570F46-566C-39FC-BEF6-635A355DD549> /usr/lib/system/libsystem_m.dylib
    0x7fff6680a000 -     0x7fff66829ff7  libsystem_malloc.dylib (140.50.7) <A45A484C-3BDE-3BC4-A219-726E3CB332D9> /usr/lib/system/libsystem_malloc.dylib
    0x7fff6682a000 -     0x7fff6695aff7  libsystem_network.dylib (1229.70.2) <5E86B2DE-9E15-3354-8714-4094ED5F698D> /usr/lib/system/libsystem_network.dylib
    0x7fff6695b000 -     0x7fff66965ffb  libsystem_networkextension.dylib (767.70.2) <9DC03712-552D-3AEE-9519-B5ED70980B70> /usr/lib/system/libsystem_networkextension.dylib
    0x7fff66966000 -     0x7fff6696fff3  libsystem_notify.dylib (172) <08012EC0-2CD2-34BE-BF93-E7F56491299A> /usr/lib/system/libsystem_notify.dylib
    0x7fff66970000 -     0x7fff66977ff7  libsystem_platform.dylib (161.50.1) <6355EE2D-5456-3CA8-A227-B96E8F1E2AF8> /usr/lib/system/libsystem_platform.dylib
    0x7fff66978000 -     0x7fff66983fff  libsystem_pthread.dylib (301.50.1) <0E51CCBA-91F2-34E1-BF2A-FEEFD3D321E4> /usr/lib/system/libsystem_pthread.dylib
    0x7fff66984000 -     0x7fff66987fff  libsystem_sandbox.dylib (765.70.1) <553DFCC6-9D31-3B9C-AB7C-30F6F265786D> /usr/lib/system/libsystem_sandbox.dylib
    0x7fff66988000 -     0x7fff66989ff3  libsystem_secinit.dylib (30) <DE8D14E8-A276-3FF8-AE13-77F7040F33C1> /usr/lib/system/libsystem_secinit.dylib
    0x7fff6698a000 -     0x7fff66991ff7  libsystem_symptoms.dylib (820.60.3) <441C6CA0-5711-3BB1-8420-DDAC3D5272E1> /usr/lib/system/libsystem_symptoms.dylib
    0x7fff66992000 -     0x7fff669a5fff  libsystem_trace.dylib (829.70.1) <3A6CB706-8CA6-3616-8AFC-14AAD7FAF187> /usr/lib/system/libsystem_trace.dylib
    0x7fff669a7000 -     0x7fff669acff7  libunwind.dylib (35.3) <BEF3FB49-5604-3B5F-82B5-332B80023AC3> /usr/lib/system/libunwind.dylib
    0x7fff669ad000 -     0x7fff669daff7  libxpc.dylib (1205.70.12) <7B82608D-ED82-35C1-B13F-99855E25D6DC> /usr/lib/system/libxpc.dylib
Sample analysis of process 28853 written to file /dev/stdout
lundman commented 4 years ago

Yeah, it shows its in 

            2745 zpool_vdev_remove  (in libzfs.2.dylib) + 336  [0x10e103068]
              2745 zfs_ioctl  (in libzfs.2.dylib) + 55  [0x10e110cd5]
                2745 ioctl  (in libsystem_kernel.dylib) + 151  [0x7fff667ad4eb]
                  2745 __ioctl  (in libsystem_kernel.dylib) + 10  [0x7fff667b33fa]

but that where we need to see the kernel stacks, yours stop exactly where it crosses into kernel.

whizzrd commented 4 years ago

Any tips on running spindump (i.e. just before starting the offending command) ? I now have 0 responsive terminal windows left

lundman commented 4 years ago

It's ok, you gave really detailed instructions, I'll give it a go in my VM tomorrow.

whizzrd commented 4 years ago

I forgot to mention that when this happened

lundman commented 4 years ago

I have tried your guide - the first problem is that cache devices do not handle /dev/diskX renumbering changes at all, and it was a bit random as to whether or not my cache device would get the same /dev/diskX as before export. When using cache/log devices, we strongly recommend to use the InvariantDisk paths, ie, /var/run/disk/ - it can even destroy non-zfs related data if some other idle disk happens to have the /dev/diskX that the pool used to think was a cache/log device.

Once I had the /dev/diskX matching I could add/remove the cache device multiple times. I see you are running 1.9.0, so it could be that we have already fixed this issue. I know the Device Removal feature had more than one fix from upstream.

whizzrd commented 4 years ago

Thanks for the repro attempt

I have tried to repro on physical and virtual environments and can not reproduce the hang using device removal.

However when testing on virtual machines (HighSierra on Virtualbox) I experience another hang (on the smb client) when using sharesmb=on on a dataset, and creating a folder using finder from another virtual smb client.

Should I create a different issue for that?

lundman commented 4 years ago

Strangely - there are no ZFS threads hung in those spindumps.