Kernel Panic: Kernel Data Abort (OpenZFS 2.1.0, M1 Ultra)

[dmzimmerman]

On my M1 Ultra with 128GB RAM, I've been getting panics like the one below.

The interesting thing about them is that what's in the backtrace is com.apple.filesystems.hfs.kext - but I'm pretty sure it's a ZFS panic, and it's just manifesting this way because I have com.apple.mimic turned on.

panic(cpu 3 caller 0xfffffe001b9a9ab0): Kernel data abort. at pc 0xfffffe001cd4ef54, lr 0xfffffe001cd4ef3c (saved state: 0xfffffe7fe1fcf6c0)
      x0:  0x0000000000000000 x1:  0x0000000000300000  x2:  0x0000000000000000  x3:  0xfffffe14cd08aca0
      x4:  0x0000000000000000 x5:  0x0000000000300000  x6:  0x0000000000000000  x7:  0xfffffe7fe1fcf868
      x8:  0x0000000000000000 x9:  0x91ffe945c10c0090  x10: 0x0000000000000000  x11: 0x0000000000000000
      x12: 0x0000000000000001 x13: 0x0000000000000014  x14: 0x0000000000044b7b  x15: 0x00000000ffffffff
      x16: 0x8bbd7e001b419780 x17: 0xfffffe001e4089d8  x18: 0x0000000000000000  x19: 0xfffffe24cc62c400
      x20: 0xfffffe2999925700 x21: 0x0000000000000000  x22: 0x0000000000300000  x23: 0x0000000000000000
      x24: 0x0000000000000000 x25: 0xfffffe7fe1fcfb20  x26: 0xfffffe001eca9920  x27: 0x0000000000000000
      x28: 0x0000000003018001 fp:  0xfffffe7fe1fcfa40  lr:  0xfffffe001cd4ef3c  sp:  0xfffffe7fe1fcfa10
      pc:  0xfffffe001cd4ef54 cpsr: 0x60401208         esr: 0x96000045          far: 0x0000000000000000

Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21G72
Kernel version: Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 5CFF5B82BEB8FAE1094CE5733AD4234D
Kernel UUID: 43BBF43C-8008-3830-8A00-C8706889EEA1
iBoot version: iBoot-7459.141.1
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x0000000013870000
KernelCache base:  0xfffffe001a874000
Kernel slide:      0x0000000014034000
Kernel text base:  0xfffffe001b038000
Kernel text exec slide: 0x000000001411c000
Kernel text exec base:  0xfffffe001b120000
mach_absolute_time: 0x15a419e072
Epoch Time:        sec       usec
  Boot    : 0x62e101c3 0x0001eef4
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x62e110da 0x00058855

Zone info:
  Zone map: 0xfffffe100040c000 - 0xfffffe300040c000
  . VM    : 0xfffffe100040c000 - 0xfffffe14cd0d8000
  . RO    : 0xfffffe14cd0d8000 - 0xfffffe1666a70000
  . GEN0  : 0xfffffe1666a70000 - 0xfffffe1b3373c000
  . GEN1  : 0xfffffe1b3373c000 - 0xfffffe2000408000
  . GEN2  : 0xfffffe2000408000 - 0xfffffe24cd0d4000
  . GEN3  : 0xfffffe24cd0d4000 - 0xfffffe2999da0000
  . DATA  : 0xfffffe2999da0000 - 0xfffffe300040c000
  Metadata: 0xfffffe8fb1dec000 - 0xfffffe8fb9dec000
  Bitmaps : 0xfffffe8fb9dec000 - 0xfffffe8fd9dec000

CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 10 PVH locks held: None
CORE 11 PVH locks held: None
CORE 12 PVH locks held: None
CORE 13 PVH locks held: None
CORE 14 PVH locks held: None
CORE 15 PVH locks held: None
CORE 16 PVH locks held: None
CORE 17 PVH locks held: None
CORE 18 PVH locks held: None
CORE 19 PVH locks held: None
CORE 0: PC=0xfffffe001b120be8, LR=0xfffffe001b2ac6fc, FP=0xfffffe7fe278ef30
CORE 1: PC=0x000000019b60fecc, LR=0x000000019e27db24, FP=0x000000016d21a420
CORE 2: PC=0xfffffe001b1af0c0, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe18aff00
CORE 3 is the one that panicked. Check the full backtrace for details.
CORE 4: PC=0xfffffe001b1af0c0, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe272ff00
CORE 5: PC=0xfffffe001b1af0c0, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe20cff00
CORE 6: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe288ff00
CORE 7: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe294ff00
CORE 8: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe87c7d6ff00
CORE 9: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe87c868ff00
CORE 10: PC=0xfffffe001b19d2f0, LR=0xfffffe001dcf972c, FP=0xfffffe87c738efb0
CORE 11: PC=0xfffffe001dcd81cc, LR=0xfffffe001dcd812c, FP=0xfffffe7fe178ef70
CORE 12: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe20eff00
CORE 13: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe87c694ff00
CORE 14: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe29aff00
CORE 15: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe87c6acff00
CORE 16: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe1e0ff00
CORE 17: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe8fafc0ff00
CORE 18: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe7fe2aaff00
CORE 19: PC=0xfffffe001b1af0bc, LR=0xfffffe001b1af0bc, FP=0xfffffe8fb0baff00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe1b33760678: 0 pages, 1375 threads: pid 0: kernel_task
Panicked thread: 0xfffffe1fffaeb0c0, backtrace: 0xfffffe7fe1fced80, tid: 2043
          lr: 0xfffffe001b179268  fp: 0xfffffe7fe1fcedf0
          lr: 0xfffffe001b178f30  fp: 0xfffffe7fe1fcee60
          lr: 0xfffffe001b2bf180  fp: 0xfffffe7fe1fcee80
          lr: 0xfffffe001b2b0f10  fp: 0xfffffe7fe1fceef0
          lr: 0xfffffe001b2aeaf4  fp: 0xfffffe7fe1fcefb0
          lr: 0xfffffe001b1277f8  fp: 0xfffffe7fe1fcefc0
          lr: 0xfffffe001b178bb4  fp: 0xfffffe7fe1fcf360
          lr: 0xfffffe001b178bb4  fp: 0xfffffe7fe1fcf3d0
          lr: 0xfffffe001b9a10e0  fp: 0xfffffe7fe1fcf3f0
          lr: 0xfffffe001b9a9ab0  fp: 0xfffffe7fe1fcf570
          lr: 0xfffffe001b2b0d10  fp: 0xfffffe7fe1fcf5e0
          lr: 0xfffffe001b2aedf8  fp: 0xfffffe7fe1fcf6a0
          lr: 0xfffffe001b1277f8  fp: 0xfffffe7fe1fcf6b0
          lr: 0xfffffe001cd4ef3c  fp: 0xfffffe7fe1fcfa40
          lr: 0xfffffe001cd7edc0  fp: 0xfffffe7fe1fcfaa0
          lr: 0xfffffe001cd3e1dc  fp: 0xfffffe7fe1fcfca0
          lr: 0xfffffe001cd41e18  fp: 0xfffffe7fe1fcfd10
          lr: 0xfffffe001cd41c2c  fp: 0xfffffe7fe1fcfd40
          lr: 0xfffffe001b3e1f80  fp: 0xfffffe7fe1fcfde0
          lr: 0xfffffe001b3e1478  fp: 0xfffffe7fe1fcfe50
          lr: 0xfffffe001b3e2ef8  fp: 0xfffffe7fe1fcfeb0
          lr: 0xfffffe001b3e2c88  fp: 0xfffffe7fe1fcff20
          lr: 0xfffffe001b130e78  fp: 0x0000000000000000
      Kernel Extensions in backtrace:
         com.apple.filesystems.hfs.kext(583.100.10)[B4C4BBD9-E1DB-3999-96C5-07C2DD3FDED1]@0xfffffe001cd38a40->0xfffffe001cd87683
            dependency: com.apple.filesystems.hfs.encodings.kext(1)[CB32F375-A103-33CB-AA27-05258BE85789]@0xfffffe001cd87690->0xfffffe001cd885bf

last started kext at 3118929074: com.apple.filesystems.smbfs    4.0 (addr 0xfffffe001b01cb50, size 65122)
loaded kexts:
org.openzfsonosx.zfs    2.1.0
com.apple.filesystems.smbfs 4.0
com.apple.driver.AppleTopCaseHIDEventDriver 5450.8
com.apple.filesystems.autofs    3.0
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL    1
com.apple.AppleEthernetAquantiaAqtionFirmware   1.0.36
com.apple.driver.AppleAHCIPort  351.100.4
com.apple.driver.usb.AppleUSBHostBillboardDevice    1.0
com.apple.driver.DiskImages.ReadWriteDiskImage  493.0.0
com.apple.driver.DiskImages.UDIFDiskImage   493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore    493.0.0
com.apple.iokit.SCSITaskUserClient  456.140.3
com.apple.driver.BCMWLANFirmware4387.Hashstore  1
com.apple.driver.AppleAOPVoiceTrigger   140.1
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleUSBDeviceNCM  5.0.0
com.apple.driver.AppleFileSystemDriver  3.0.1
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.nke.l2tp  1.9
com.apple.filesystems.tmpfs 1
com.apple.filesystems.lifs  1
com.apple.filesystems.apfs  1934.141.2
com.apple.IOTextEncryptionFamily    1.0.0
com.apple.filesystems.hfs.kext  583.100.10
com.apple.security.BootPolicy   1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleEmbeddedSimpleSPINORFlasher  1
com.apple.driver.AppleSmartIO2  1
com.apple.driver.ApplePMP   1
com.apple.driver.AppleSN012776Amp   550.3
com.apple.driver.AppleCS42L84Audio  550.3
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6002CLPC 1
com.apple.driver.AppleSmartBatteryManager   161.0.0
com.apple.driver.ApplePMPFirmware   1
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AppleS5L8920XPWM   1.0.0d1
com.apple.driver.AppleJPEGDriver    4.8.1
com.apple.driver.AudioDMAController-T600x   160.2
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleS5L8960XNCO   1
com.apple.driver.AppleS8000AES  1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim    1
com.apple.driver.AppleProResHW  177.0.0
com.apple.AGXG13X   190.22
com.apple.driver.AppleAVE2  560.5.0
com.apple.driver.AppleAVD   566
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleSDXC  3.2.1
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleMobileDispT600X-DCP   140.0
com.apple.driver.AppleSPMIPMU   1.0.1
com.apple.driver.AppleEventLogHandler   1
com.apple.driver.AppleS8000DWI  1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule   1
com.apple.driver.AppleBCMWLANBusInterfacePCIe   1
com.apple.driver.AppleS5L8940XI2C   1.0.0d2
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleT6000 1
com.apple.driver.AppleM68Buttons    1.0.0d1
com.apple.iokit.IOUserEthernet  1.0.1
com.apple.driver.usb.AppleUSBUserHCI    1
com.apple.iokit.IOKitRegistryCompatibility  1
com.apple.iokit.EndpointSecurity    1
com.apple.driver.AppleDiskImages2   126.141.2
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall   402
com.apple.kec.InvalidateHmac    1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial   6.0.0
com.apple.driver.AppleActuatorDriver    5460.1
com.apple.driver.AppleMultitouchDriver  5460.1
com.apple.driver.AppleInputDeviceSupport    5460.1
com.apple.driver.AppleHSBluetoothDriver 5450.8
com.apple.driver.IOBluetoothHIDDriver   9.0.0
com.apple.iokit.IOAVBFamily 1040.6
com.apple.plugin.IOgPTPPlugin   1040.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.iokit.IOAHCIBlockStorage  333.140.2
com.apple.kext.triggers 1.0
com.apple.driver.AppleUSBMergeNub   900.4.2
com.apple.driver.usb.cdc.ecm    5.0.0
com.apple.driver.usb.cdc.acm    5.0.0
com.apple.driver.usb.serial 6.0.0
com.apple.driver.usb.cdc.ncm    5.0.0
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily   1
com.apple.iokit.IOAHCIFamily    297
com.apple.driver.AppleEthernetAquantiaAqtion    1.0.64
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.AppleThunderboltPCIUpAdapter   4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter   8.5.1
com.apple.driver.DiskImages.KernelBacked    493.0.0
com.apple.driver.usb.cdc    5.0.0
com.apple.driver.AppleUSBAudio  416.2
com.apple.iokit.IOAudioFamily   340.2
com.apple.vecLib.kext   1.2.0
com.apple.driver.AppleXsanScheme    3
com.apple.driver.AppleAOPAudio  150.2
com.apple.driver.AppleSEPHDCPManager    1.0.1
com.apple.driver.AppleTrustedAccessory  1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI    1
com.apple.driver.usb.AppleUSBXHCIPCI    1.2
com.apple.driver.AppleEmbeddedUSB   1
com.apple.driver.AppleConvergedIPCOLYBTControl  1
com.apple.driver.AppleConvergedPCI  1
com.apple.driver.AppleBluetoothDebug    1
com.apple.driver.usb.networking 5.0.0
com.apple.nke.ppp   1.9
com.apple.driver.AppleBSDKextStarter    3
com.apple.filesystems.hfs.encodings.kext    1
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltDPInAdapter    8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily    8.5.1
com.apple.driver.AppleDiagnosticDataAccessReadOnly  1.0.0
com.apple.driver.AppleBTM   1.0.1
com.apple.driver.AppleI2CEthernetAquantia   1.0.0
com.apple.driver.AppleDCPDPTXProxy  1.0.0
com.apple.driver.DCPDPFamilyProxy   1
com.apple.driver.AppleHPM   3.4.4
com.apple.driver.AppleCSEmbeddedAudio   550.3
com.apple.driver.AppleEmbeddedAudio 550.3
com.apple.iokit.AppleARMIISAudio    140.2
com.apple.driver.ApplePassthroughPPM    3.0
com.apple.driver.usb.AppleUSBHostPacketFilter   1.0
com.apple.AGXFirmwareKextG13XRTBuddy    190.22
com.apple.AGXFirmwareKextRTBuddy64  190.22
com.apple.iokit.IONVMeFamily    2.1.0
com.apple.driver.AppleNANDConfigAccess  1.0.0
com.apple.driver.AppleStockholmControl  1.0.0
com.apple.driver.DCPAVFamilyProxy   1
com.apple.driver.AppleMCA2-T600x    640.5
com.apple.driver.AppleEmbeddedAudioLibs 140.1
com.apple.driver.ApplePMGR  1
com.apple.driver.AppleSPU   1
com.apple.iokit.IOGPUFamily 35.29
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.driver.AppleUSBXDCIARM    1.0
com.apple.driver.AppleUSBXDCI   1.0
com.apple.iokit.IOUSBDeviceFamily   2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI   1
com.apple.driver.usb.AppleUSBXHCI   1.2
com.apple.driver.AppleEmbeddedUSBHost   1
com.apple.driver.usb.AppleUSBHub    1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice    1.2
com.apple.iokit.IOMobileGraphicsFamily-DCP  343.0.0
com.apple.driver.AppleDCP   1
com.apple.driver.AppleFirmwareKit   1
com.apple.driver.AppleFirmwareUpdateKext    1
com.apple.iokit.IOMobileGraphicsFamily  343.0.0
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleSPMI  1.0.1
com.apple.driver.AppleSART  1
com.apple.driver.AppleARMWatchdogTimer  1
com.apple.driver.AppleDisplayCrossbar   1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy  1
com.apple.driver.AppleThunderboltNHI    7.2.81
com.apple.driver.AppleT6000PCIeC    1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA    1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleMultiFunctionManager  1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore   1.0.0
com.apple.iokit.IO80211Family   1200.12.2b2
com.apple.driver.IOImageLoader  1.0.0
com.apple.driver.AppleOLYHAL    1
com.apple.driver.corecapture    1.0.4
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor 1.0.0
com.apple.driver.AppleEmbeddedPCIE  1
com.apple.driver.AppleGPIOICController  1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMobileApNonce 1
com.apple.iokit.IOTimeSyncFamily    1040.3
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily    597
com.apple.iokit.IOBluetoothSerialManager    9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport   9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport  9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport  9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily   9.0.0
com.apple.driver.FairPlayIOKit  68.16.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport  9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE   1.0
com.apple.driver.AppleSEPKeyStore   2
com.apple.driver.AppleUSBTDM    533.120.2
com.apple.iokit.IOUSBMassStorageDriver  210.120.3
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice   456.140.3
com.apple.iokit.IOSCSIArchitectureModelFamily   456.140.3
com.apple.driver.AppleIPAppender    1.0
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost    1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties    1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC   3.1.9
com.apple.driver.RTBuddy    1.0.0
com.apple.driver.AppleEmbeddedTempSensor    1.0.0
com.apple.driver.AppleARMPMU    1.0
com.apple.iokit.IOAccessoryManager  1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient  1.0.1b8
com.apple.iokit.IONetworkingFamily  3.4
com.apple.iokit.IOSerialFamily  11
com.apple.driver.AppleSEPManager    1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor   1
com.apple.driver.AppleBiometricSensor   2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.driver.AppleANELoadBalancer   5.68.2
com.apple.driver.AppleH11ANEInterface   5.68.0
com.apple.AUC   1.0
com.apple.iokit.IOAVFamily  1.0.0
com.apple.iokit.IOHDCPFamily    1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family  1.0
com.apple.driver.AppleIISController 140.1
com.apple.driver.AppleAudioClockLibs    140.1
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface   302.14
com.apple.driver.IODARTFamily   1
com.apple.security.quarantine   4
com.apple.security.sandbox  300.0
com.apple.kext.AppleMatch   1.0.0d1
com.apple.driver.AppleMobileFileIntegrity   1.0.5
com.apple.security.AppleImage4  4.2.0
com.apple.kext.CoreTrust    1
com.apple.iokit.IOCryptoAcceleratorFamily   1.0.1
com.apple.driver.AppleARMPlatform   1.0.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily    1.0.0
com.apple.iokit.IOReportFamily  47
com.apple.kec.pthread   1
com.apple.kec.Libm  1
com.apple.kec.corecrypto    12.0

** Stackshot Succeeded ** Bytes Traced 342779 (Uncompressed 1006768) **

[dmzimmerman]

Another one:

panic(cpu 2 caller 0xfffffe00159a9ab0): Kernel data abort. at pc 0xfffffe0016d4ef54, lr 0xfffffe0016d4ef3c (saved state: 0xfffffe686beef510)
      x0:  0x0000000000000000 x1:  0x0000000000300000  x2:  0x0000000000000000  x3:  0xfffffe1000542ca0
      x4:  0x0000000000000000 x5:  0x0000000000300000  x6:  0x0000000000000000  x7:  0xfffffe686beef6b8
      x8:  0x0000000000000000 x9:  0x2f7cb76c5cdb00c5  x10: 0x0000000000000000  x11: 0x0000000000000000
      x12: 0x0000000000000001 x13: 0x0000000000000014  x14: 0x000000000003726c  x15: 0x00000000ffffffff
      x16: 0x4c9c7e0015419780 x17: 0xfffffe00184089d8  x18: 0x0000000000000000  x19: 0xfffffe299942ac00
      x20: 0xfffffe2001226a80 x21: 0x0000000000000000  x22: 0x0000000000300000  x23: 0x0000000000000000
      x24: 0x0000000000000000 x25: 0xfffffe686beef970  x26: 0xfffffe0018ca9920  x27: 0x0000000000000000
      x28: 0x0000000003018001 fp:  0xfffffe686beef890  lr:  0xfffffe0016d4ef3c  sp:  0xfffffe686beef860
      pc:  0xfffffe0016d4ef54 cpsr: 0x60401208         esr: 0x96000046          far: 0x0000000000000000

Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21G72
Kernel version: Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 5CFF5B82BEB8FAE1094CE5733AD4234D
Kernel UUID: 43BBF43C-8008-3830-8A00-C8706889EEA1
iBoot version: iBoot-7459.141.1
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x000000000d870000
KernelCache base:  0xfffffe0014874000
Kernel slide:      0x000000000e034000
Kernel text base:  0xfffffe0015038000
Kernel text exec slide: 0x000000000e11c000
Kernel text exec base:  0xfffffe0015120000
mach_absolute_time: 0x2fec9d56c1
Epoch Time:        sec       usec
  Boot    : 0x62e23e3d 0x000d8bf1
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x62e25fb4 0x00041d68

Zone info:
  Zone map: 0xfffffe1000524000 - 0xfffffe3000524000
  . VM    : 0xfffffe1000524000 - 0xfffffe14cd1f0000
  . RO    : 0xfffffe14cd1f0000 - 0xfffffe1666b88000
  . GEN0  : 0xfffffe1666b88000 - 0xfffffe1b33854000
  . GEN1  : 0xfffffe1b33854000 - 0xfffffe2000520000
  . GEN2  : 0xfffffe2000520000 - 0xfffffe24cd1ec000
  . GEN3  : 0xfffffe24cd1ec000 - 0xfffffe2999eb8000
  . DATA  : 0xfffffe2999eb8000 - 0xfffffe3000524000
  Metadata: 0xfffffe3000534000 - 0xfffffe3008534000
  Bitmaps : 0xfffffe3008534000 - 0xfffffe3028534000

CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 10 PVH locks held: None
CORE 11 PVH locks held: None
CORE 12 PVH locks held: None
CORE 13 PVH locks held: None
CORE 14 PVH locks held: None
CORE 15 PVH locks held: None
CORE 16 PVH locks held: None
CORE 17 PVH locks held: None
CORE 18 PVH locks held: None
CORE 19 PVH locks held: None
CORE 0: PC=0xfffffe00151dfac4, LR=0xfffffe00146d87d0, FP=0xfffffe608b64df10
CORE 1: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686db2ff00
CORE 2 is the one that panicked. Check the full backtrace for details.
CORE 3: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe687292ff00
CORE 4: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686b9cff00
CORE 5: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686bacff00
CORE 6: PC=0xfffffe00152b3df8, LR=0xfffffe00152b3df4, FP=0xfffffe686ce2fe90
CORE 7: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686bbcff00
CORE 8: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe608b6cff00
CORE 9: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686becff00
CORE 10: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686c2aff00
CORE 11: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe608b76ff00
CORE 12: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686c0eff00
CORE 13: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686bf0ff00
CORE 14: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686cf8ff00
CORE 15: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686c18ff00
CORE 16: PC=0xfffffe00152b3df8, LR=0xfffffe00152b3df4, FP=0xfffffe686cecfe90
CORE 17: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686ec2ff00
CORE 18: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe60883aff00
CORE 19: PC=0xfffffe00151af0bc, LR=0xfffffe00151af0bc, FP=0xfffffe686ce0ff00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe1ffeb8b470: 101 pages, 1 threads: pid 3097: rsync
Panicked thread: 0xfffffe24cb7fa8a0, backtrace: 0xfffffe686beeebd0, tid: 68762
          lr: 0xfffffe0015179268  fp: 0xfffffe686beeec40
          lr: 0xfffffe0015178f30  fp: 0xfffffe686beeecb0
          lr: 0xfffffe00152bf180  fp: 0xfffffe686beeecd0
          lr: 0xfffffe00152b0f10  fp: 0xfffffe686beeed40
          lr: 0xfffffe00152aeaf4  fp: 0xfffffe686beeee00
          lr: 0xfffffe00151277f8  fp: 0xfffffe686beeee10
          lr: 0xfffffe0015178bb4  fp: 0xfffffe686beef1b0
          lr: 0xfffffe0015178bb4  fp: 0xfffffe686beef220
          lr: 0xfffffe00159a10e0  fp: 0xfffffe686beef240
          lr: 0xfffffe00159a9ab0  fp: 0xfffffe686beef3c0
          lr: 0xfffffe00152b0d10  fp: 0xfffffe686beef430
          lr: 0xfffffe00152aedf8  fp: 0xfffffe686beef4f0
          lr: 0xfffffe00151277f8  fp: 0xfffffe686beef500
          lr: 0xfffffe0016d4ef3c  fp: 0xfffffe686beef890
          lr: 0xfffffe0016d7edc0  fp: 0xfffffe686beef8f0
          lr: 0xfffffe0016d3e1dc  fp: 0xfffffe686beefaf0
          lr: 0xfffffe0016d41e18  fp: 0xfffffe686beefb60
          lr: 0xfffffe0016d41c2c  fp: 0xfffffe686beefb90
          lr: 0xfffffe001540eb64  fp: 0xfffffe686beefc20
          lr: 0xfffffe001540ea18  fp: 0xfffffe686beefc80
          lr: 0xfffffe001568c038  fp: 0xfffffe686beefce0
          lr: 0xfffffe001568cf60  fp: 0xfffffe686beefdb0
          lr: 0xfffffe00157e2614  fp: 0xfffffe686beefe50
          lr: 0xfffffe00152aebc8  fp: 0xfffffe686beeff10
          lr: 0xfffffe00151277f8  fp: 0xfffffe686beeff20
      Kernel Extensions in backtrace:
         com.apple.filesystems.hfs.kext(583.100.10)[B4C4BBD9-E1DB-3999-96C5-07C2DD3FDED1]@0xfffffe0016d38a40->0xfffffe0016d87683
            dependency: com.apple.filesystems.hfs.encodings.kext(1)[CB32F375-A103-33CB-AA27-05258BE85789]@0xfffffe0016d87690->0xfffffe0016d885bf

last started kext at 2084133148: com.apple.filesystems.smbfs    4.0 (addr 0xfffffe001501cb50, size 65122)
loaded kexts:
org.openzfsonosx.zfs    2.1.0
com.apple.filesystems.smbfs 4.0
com.apple.driver.AppleTopCaseHIDEventDriver 5450.8
com.apple.filesystems.autofs    3.0
com.apple.AppleEthernetAquantiaAqtionFirmware   1.0.36
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL    1
com.apple.driver.AppleAHCIPort  351.100.4
com.apple.driver.usb.AppleUSBHostBillboardDevice    1.0
com.apple.driver.DiskImages.ReadWriteDiskImage  493.0.0
com.apple.driver.DiskImages.UDIFDiskImage   493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore    493.0.0
com.apple.iokit.SCSITaskUserClient  456.140.3
com.apple.driver.BCMWLANFirmware4387.Hashstore  1
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleUSBDeviceNCM  5.0.0
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleFileSystemDriver  3.0.1
com.apple.nke.l2tp  1.9
com.apple.filesystems.tmpfs 1
com.apple.filesystems.lifs  1
com.apple.filesystems.apfs  1934.141.2
com.apple.IOTextEncryptionFamily    1.0.0
com.apple.filesystems.hfs.kext  583.100.10
com.apple.security.BootPolicy   1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.driver.AppleSmartIO2  1
com.apple.driver.ApplePMP   1
com.apple.AppleEmbeddedSimpleSPINORFlasher  1
com.apple.driver.AppleSN012776Amp   550.3
com.apple.driver.AppleCS42L84Audio  550.3
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6002CLPC 1
com.apple.driver.AppleSmartBatteryManager   161.0.0
com.apple.driver.AppleAOPVoiceTrigger   140.1
com.apple.driver.ApplePMPFirmware   1
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AppleS5L8920XPWM   1.0.0d1
com.apple.driver.AppleJPEGDriver    4.8.1
com.apple.driver.AudioDMAController-T600x   160.2
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleS5L8960XNCO   1
com.apple.driver.AppleS8000AES  1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim    1
com.apple.driver.AppleProResHW  177.0.0
com.apple.AGXG13X   190.22
com.apple.driver.AppleAVE2  560.5.0
com.apple.driver.AppleAVD   566
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleSDXC  3.2.1
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleMobileDispT600X-DCP   140.0
com.apple.driver.AppleSPMIPMU   1.0.1
com.apple.driver.AppleEventLogHandler   1
com.apple.driver.AppleS8000DWI  1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule   1
com.apple.driver.AppleBCMWLANBusInterfacePCIe   1
com.apple.driver.AppleS5L8940XI2C   1.0.0d2
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleT6000 1
com.apple.driver.AppleM68Buttons    1.0.0d1
com.apple.iokit.IOUserEthernet  1.0.1
com.apple.driver.usb.AppleUSBUserHCI    1
com.apple.iokit.IOKitRegistryCompatibility  1
com.apple.iokit.EndpointSecurity    1
com.apple.driver.AppleDiskImages2   126.141.2
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall   402
com.apple.kec.InvalidateHmac    1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial   6.0.0
com.apple.driver.AppleActuatorDriver    5460.1
com.apple.iokit.IOAVBFamily 1040.6
com.apple.plugin.IOgPTPPlugin   1040.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.driver.AppleMultitouchDriver  5460.1
com.apple.driver.AppleInputDeviceSupport    5460.1
com.apple.driver.AppleHSBluetoothDriver 5450.8
com.apple.driver.IOBluetoothHIDDriver   9.0.0
com.apple.iokit.IOAHCIBlockStorage  333.140.2
com.apple.kext.triggers 1.0
com.apple.driver.AppleUSBMergeNub   900.4.2
com.apple.driver.usb.cdc.ncm    5.0.0
com.apple.driver.usb.cdc.ecm    5.0.0
com.apple.driver.usb.cdc.acm    5.0.0
com.apple.driver.usb.serial 6.0.0
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily   1
com.apple.iokit.IOAHCIFamily    297
com.apple.driver.AppleThunderboltPCIUpAdapter   4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter   8.5.1
com.apple.driver.AppleEthernetAquantiaAqtion    1.0.64
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.DiskImages.KernelBacked    493.0.0
com.apple.driver.usb.cdc    5.0.0
com.apple.driver.AppleUSBAudio  416.2
com.apple.iokit.IOAudioFamily   340.2
com.apple.vecLib.kext   1.2.0
com.apple.driver.AppleXsanScheme    3
com.apple.driver.AppleSEPHDCPManager    1.0.1
com.apple.driver.AppleTrustedAccessory  1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI    1
com.apple.driver.usb.AppleUSBXHCIPCI    1.2
com.apple.driver.AppleEmbeddedUSB   1
com.apple.driver.AppleConvergedIPCOLYBTControl  1
com.apple.driver.AppleConvergedPCI  1
com.apple.driver.AppleBluetoothDebug    1
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltDPInAdapter    8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily    8.5.1
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.nke.ppp   1.9
com.apple.driver.AppleBSDKextStarter    3
com.apple.filesystems.hfs.encodings.kext    1
com.apple.driver.AppleBTM   1.0.1
com.apple.driver.AppleDiagnosticDataAccessReadOnly  1.0.0
com.apple.driver.AppleDCPDPTXProxy  1.0.0
com.apple.driver.DCPDPFamilyProxy   1
com.apple.driver.AppleHPM   3.4.4
com.apple.driver.AppleI2CEthernetAquantia   1.0.0
com.apple.driver.AppleCSEmbeddedAudio   550.3
com.apple.driver.ApplePassthroughPPM    3.0
com.apple.driver.usb.AppleUSBHostPacketFilter   1.0
com.apple.driver.AppleAOPAudio  150.2
com.apple.driver.AppleEmbeddedAudio 550.3
com.apple.iokit.AppleARMIISAudio    140.2
com.apple.AGXFirmwareKextG13XRTBuddy    190.22
com.apple.AGXFirmwareKextRTBuddy64  190.22
com.apple.driver.DCPAVFamilyProxy   1
com.apple.iokit.IONVMeFamily    2.1.0
com.apple.driver.AppleNANDConfigAccess  1.0.0
com.apple.driver.AppleMCA2-T600x    640.5
com.apple.driver.AppleEmbeddedAudioLibs 140.1
com.apple.driver.ApplePMGR  1
com.apple.driver.AppleSPU   1
com.apple.iokit.IOGPUFamily 35.29
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.driver.AppleUSBXDCIARM    1.0
com.apple.driver.AppleUSBXDCI   1.0
com.apple.iokit.IOUSBDeviceFamily   2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI   1
com.apple.driver.usb.AppleUSBXHCI   1.2
com.apple.driver.AppleEmbeddedUSBHost   1
com.apple.driver.usb.AppleUSBHub    1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice    1.2
com.apple.driver.AppleStockholmControl  1.0.0
com.apple.iokit.IOMobileGraphicsFamily-DCP  343.0.0
com.apple.driver.AppleDCP   1
com.apple.driver.AppleFirmwareKit   1
com.apple.driver.AppleFirmwareUpdateKext    1
com.apple.iokit.IOMobileGraphicsFamily  343.0.0
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleSPMI  1.0.1
com.apple.driver.AppleSART  1
com.apple.driver.AppleARMWatchdogTimer  1
com.apple.driver.AppleDisplayCrossbar   1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy  1
com.apple.driver.AppleThunderboltNHI    7.2.81
com.apple.driver.AppleT6000PCIeC    1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA    1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleMultiFunctionManager  1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore   1.0.0
com.apple.iokit.IO80211Family   1200.12.2b2
com.apple.driver.IOImageLoader  1.0.0
com.apple.driver.AppleOLYHAL    1
com.apple.driver.corecapture    1.0.4
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor 1.0.0
com.apple.driver.AppleEmbeddedPCIE  1
com.apple.driver.AppleGPIOICController  1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMobileApNonce 1
com.apple.iokit.IOTimeSyncFamily    1040.3
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily    597
com.apple.iokit.IOBluetoothSerialManager    9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport   9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport  9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport  9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily   9.0.0
com.apple.driver.FairPlayIOKit  68.16.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport  9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE   1.0
com.apple.driver.AppleSEPKeyStore   2
com.apple.driver.AppleUSBTDM    533.120.2
com.apple.iokit.IOUSBMassStorageDriver  210.120.3
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice   456.140.3
com.apple.iokit.IOSCSIArchitectureModelFamily   456.140.3
com.apple.driver.AppleIPAppender    1.0
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost    1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties    1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC   3.1.9
com.apple.driver.RTBuddy    1.0.0
com.apple.driver.AppleEmbeddedTempSensor    1.0.0
com.apple.driver.AppleARMPMU    1.0
com.apple.iokit.IOAccessoryManager  1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient  1.0.1b8
com.apple.iokit.IONetworkingFamily  3.4
com.apple.iokit.IOSerialFamily  11
com.apple.driver.AppleSEPManager    1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor   1
com.apple.driver.AppleBiometricSensor   2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.driver.AppleANELoadBalancer   5.68.2
com.apple.driver.AppleH11ANEInterface   5.68.0
com.apple.AUC   1.0
com.apple.iokit.IOAVFamily  1.0.0
com.apple.iokit.IOHDCPFamily    1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family  1.0
com.apple.driver.AppleIISController 140.1
com.apple.driver.AppleAudioClockLibs    140.1
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface   302.14
com.apple.driver.IODARTFamily   1
com.apple.security.quarantine   4
com.apple.security.sandbox  300.0
com.apple.kext.AppleMatch   1.0.0d1
com.apple.driver.AppleMobileFileIntegrity   1.0.5
com.apple.security.AppleImage4  4.2.0
com.apple.kext.CoreTrust    1
com.apple.iokit.IOCryptoAcceleratorFamily   1.0.1
com.apple.driver.AppleARMPlatform   1.0.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily    1.0.0
com.apple.iokit.IOReportFamily  47
com.apple.kec.pthread   1
com.apple.kec.Libm  1
com.apple.kec.corecrypto    12.0

** Stackshot Succeeded ** Bytes Traced 342095 (Uncompressed 1005120) **

[dmzimmerman]

Another one (different process panicking this time):

panic(cpu 1 caller 0xfffffe00299a9ab0): Kernel data abort. at pc 0xfffffe00292ad71c, lr 0x6293fe00292ad6c0 (saved state: 0xfffffe7fde02f080)
      x0:  0x0000000000000000 x1:  0xfffffe166d5daae0  x2:  0xfffffe7fdbc84000  x3:  0x000000000000c000
      x4:  0xfffffe14d3b92cd0 x5:  0x0000000100000000  x6:  0xfffffe7fde02f108  x7:  0xfffffe7fde02f100
      x8:  0xfffffe002cec5000 x9:  0xfffffe002cc65000  x10: 0x0000000000000000  x11: 0x0000000000000000
      x12: 0x0000000000000000 x13: 0xfffffe87ec0bc000  x14: 0x0000000000000001  x15: 0x00000000000000a8
      x16: 0x89c9fe00292ad6a4 x17: 0xfffffe002c8cae20  x18: 0x0000000000000000  x19: 0x0000000000000000
      x20: 0xfffffe166d27bc00 x21: 0xfffffe200785de60  x22: 0xfffffe166dc455c0  x23: 0xfffffe1b3cb24000
      x24: 0x0000000000000000 x25: 0xfffffe002cec5000  x26: 0x0000000000000001  x27: 0x0000000000000000
      x28: 0x0000000000000000 fp:  0xfffffe7fde02f3f0  lr:  0x6293fe00292ad6c0  sp:  0xfffffe7fde02f3d0
      pc:  0xfffffe00292ad71c cpsr: 0x60401208         esr: 0x96000006          far: 0x000000000000000b

Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21G72
Kernel version: Darwin Kernel Version 21.6.0: Sat Jun 18 17:07:22 PDT 2022; root:xnu-8020.140.41~1/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 5CFF5B82BEB8FAE1094CE5733AD4234D
Kernel UUID: 43BBF43C-8008-3830-8A00-C8706889EEA1
iBoot version: iBoot-7459.141.1
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x0000000021870000
KernelCache base:  0xfffffe0028874000
Kernel slide:      0x0000000022034000
Kernel text base:  0xfffffe0029038000
Kernel text exec slide: 0x000000002211c000
Kernel text exec base:  0xfffffe0029120000
mach_absolute_time: 0x42edf334fb
Epoch Time:        sec       usec
  Boot    : 0x62e77d5a 0x0000f05d
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x62e7ac19 0x000c5d6b

Zone info:
  Zone map: 0xfffffe1006f1c000 - 0xfffffe3006f1c000
  . VM    : 0xfffffe1006f1c000 - 0xfffffe14d3be8000
  . RO    : 0xfffffe14d3be8000 - 0xfffffe166d580000
  . GEN0  : 0xfffffe166d580000 - 0xfffffe1b3a24c000
  . GEN1  : 0xfffffe1b3a24c000 - 0xfffffe2006f18000
  . GEN2  : 0xfffffe2006f18000 - 0xfffffe24d3be4000
  . GEN3  : 0xfffffe24d3be4000 - 0xfffffe29a08b0000
  . DATA  : 0xfffffe29a08b0000 - 0xfffffe3006f1c000
  Metadata: 0xfffffe87e40bc000 - 0xfffffe87ec0bc000
  Bitmaps : 0xfffffe87ec0bc000 - 0xfffffe880c0bc000

CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 10 PVH locks held: None
CORE 11 PVH locks held: None
CORE 12 PVH locks held: None
CORE 13 PVH locks held: None
CORE 14 PVH locks held: None
CORE 15 PVH locks held: None
CORE 16 PVH locks held: None
CORE 17 PVH locks held: None
CORE 18 PVH locks held: None
CORE 19 PVH locks held: None
CORE 0: PC=0x00007ff899cb007c, LR=0x00007ff89bd3a0ac, FP=0x00000002048fa6e0
CORE 1 is the one that panicked. Check the full backtrace for details.
CORE 2: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe7fddccff00
CORE 3: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe8ff674ff00
CORE 4: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fde08ff00
CORE 5: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe8810d4ff00
CORE 6: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe87c2f2ff00
CORE 7: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe8ff80cff00
CORE 8: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe7fddbeff00
CORE 9: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe7fdda8ff00
CORE 10: PC=0xfffffe00291dfcb0, LR=0xfffffe00291dfcb0, FP=0xfffffe7fdd44f980
CORE 11: PC=0x000000019e05a0d4, LR=0x000000019e1fed8c, FP=0x000000016b0d5cc0
CORE 12: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fdd26ff00
CORE 13: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fdcceff00
CORE 14: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fdd4eff00
CORE 15: PC=0xfffffe00292b3df8, LR=0xfffffe00292b3df4, FP=0xfffffe8ff6c0fe90
CORE 16: PC=0xfffffe00291af0c0, LR=0xfffffe00291af0bc, FP=0xfffffe881060ff00
CORE 17: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe87c152ff00
CORE 18: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe7fdc78ff00
CORE 19: PC=0xfffffe00291af0bc, LR=0xfffffe00291af0bc, FP=0xfffffe8ff948ff00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe1b36f56df8: 804 pages, 6 threads: pid 24350: NetAuthSysAgent
Panicked thread: 0xfffffe24d572c820, backtrace: 0xfffffe7fde02e740, tid: 204587
          lr: 0xfffffe0029179268  fp: 0xfffffe7fde02e7b0
          lr: 0xfffffe0029178f30  fp: 0xfffffe7fde02e820
          lr: 0xfffffe00292bf180  fp: 0xfffffe7fde02e840
          lr: 0xfffffe00292b0f10  fp: 0xfffffe7fde02e8b0
          lr: 0xfffffe00292aeaf4  fp: 0xfffffe7fde02e970
          lr: 0xfffffe00291277f8  fp: 0xfffffe7fde02e980
          lr: 0xfffffe0029178bb4  fp: 0xfffffe7fde02ed20
          lr: 0xfffffe0029178bb4  fp: 0xfffffe7fde02ed90
          lr: 0xfffffe00299a10e0  fp: 0xfffffe7fde02edb0
          lr: 0xfffffe00299a9ab0  fp: 0xfffffe7fde02ef30
          lr: 0xfffffe00292b0d10  fp: 0xfffffe7fde02efa0
          lr: 0xfffffe00292aec64  fp: 0xfffffe7fde02f060
          lr: 0xfffffe00291277f8  fp: 0xfffffe7fde02f070
          lr: 0xfffffe00292ad6c0  fp: 0xfffffe7fde02f3f0
          lr: 0xfffffe00292ad6c0  fp: 0xfffffe7fde02f420
          lr: 0xfffffe002be23484  fp: 0xfffffe7fde02f6e0
          lr: 0xfffffe00293f1234  fp: 0xfffffe7fde02f970
          lr: 0xfffffe00293f2a20  fp: 0xfffffe7fde02fd70
          lr: 0xfffffe00293f26d8  fp: 0xfffffe7fde02fdb0
          lr: 0xfffffe00297e2614  fp: 0xfffffe7fde02fe50
          lr: 0xfffffe00292aebc8  fp: 0xfffffe7fde02ff10
          lr: 0xfffffe00291277f8  fp: 0xfffffe7fde02ff20
      Kernel Extensions in backtrace:
         com.apple.filesystems.smbfs(4.0)[C35FF09E-CA77-3337-9F5B-39538E87FF32]@0xfffffe002be0d890->0xfffffe002be70c93
            dependency: com.apple.kec.corecrypto(12.0)[CD18ACA5-5618-3CCF-9232-A8592E5CF16E]@0xfffffe002bd52000->0xfffffe002bd9e877
            dependency: com.apple.kext.triggers(1.0)[7C659EAA-EEC9-3109-8DB4-E93FB895DEDD]@0xfffffe002be7acf0->0xfffffe002be7da37

last started kext at 3075925156: com.apple.filesystems.smbfs    4.0 (addr 0xfffffe002901cb50, size 65122)
loaded kexts:
org.openzfsonosx.zfs    2.1.0
com.apple.filesystems.smbfs 4.0
com.apple.driver.AppleTopCaseHIDEventDriver 5450.8
com.apple.filesystems.autofs    3.0
com.apple.AppleEthernetAquantiaAqtionFirmware   1.0.36
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL    1
com.apple.driver.AppleAHCIPort  351.100.4
com.apple.driver.usb.AppleUSBHostBillboardDevice    1.0
com.apple.driver.DiskImages.ReadWriteDiskImage  493.0.0
com.apple.driver.DiskImages.UDIFDiskImage   493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore    493.0.0
com.apple.iokit.SCSITaskUserClient  456.140.3
com.apple.driver.BCMWLANFirmware4387.Hashstore  1
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleUSBDeviceNCM  5.0.0
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleFileSystemDriver  3.0.1
com.apple.nke.l2tp  1.9
com.apple.filesystems.tmpfs 1
com.apple.filesystems.lifs  1
com.apple.filesystems.apfs  1934.141.2
com.apple.IOTextEncryptionFamily    1.0.0
com.apple.filesystems.hfs.kext  583.100.10
com.apple.security.BootPolicy   1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleEmbeddedSimpleSPINORFlasher  1
com.apple.driver.ApplePMP   1
com.apple.driver.AppleSN012776Amp   550.3
com.apple.driver.AppleSmartIO2  1
com.apple.driver.AppleCS42L84Audio  550.3
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6002CLPC 1
com.apple.driver.AppleSmartBatteryManager   161.0.0
com.apple.driver.AppleAOPVoiceTrigger   140.1
com.apple.driver.ApplePMPFirmware   1
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AppleS5L8920XPWM   1.0.0d1
com.apple.driver.AppleJPEGDriver    4.8.1
com.apple.driver.AudioDMAController-T600x   160.2
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleS5L8960XNCO   1
com.apple.driver.AppleS8000AES  1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim    1
com.apple.driver.AppleProResHW  177.0.0
com.apple.AGXG13X   190.22
com.apple.driver.AppleAVE2  560.5.0
com.apple.driver.AppleAVD   566
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleSDXC  3.2.1
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleMobileDispT600X-DCP   140.0
com.apple.driver.AppleSPMIPMU   1.0.1
com.apple.driver.AppleEventLogHandler   1
com.apple.driver.AppleS8000DWI  1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule   1
com.apple.driver.AppleBCMWLANBusInterfacePCIe   1
com.apple.driver.AppleS5L8940XI2C   1.0.0d2
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleT6000 1
com.apple.driver.AppleM68Buttons    1.0.0d1
com.apple.iokit.IOUserEthernet  1.0.1
com.apple.driver.usb.AppleUSBUserHCI    1
com.apple.iokit.IOKitRegistryCompatibility  1
com.apple.iokit.EndpointSecurity    1
com.apple.driver.AppleDiskImages2   126.141.2
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall   402
com.apple.kec.InvalidateHmac    1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial   6.0.0
com.apple.iokit.IOAVBFamily 1040.6
com.apple.plugin.IOgPTPPlugin   1040.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.iokit.IOAHCIBlockStorage  333.140.2
com.apple.driver.AppleActuatorDriver    5460.1
com.apple.driver.AppleMultitouchDriver  5460.1
com.apple.driver.AppleInputDeviceSupport    5460.1
com.apple.driver.AppleHSBluetoothDriver 5450.8
com.apple.driver.IOBluetoothHIDDriver   9.0.0
com.apple.kext.triggers 1.0
com.apple.driver.AppleUSBMergeNub   900.4.2
com.apple.driver.usb.cdc.ncm    5.0.0
com.apple.driver.usb.cdc.ecm    5.0.0
com.apple.driver.usb.cdc.acm    5.0.0
com.apple.driver.usb.serial 6.0.0
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily   1
com.apple.iokit.IOAHCIFamily    297
com.apple.driver.AppleEthernetAquantiaAqtion    1.0.64
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.AppleThunderboltPCIUpAdapter   4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter   8.5.1
com.apple.driver.DiskImages.KernelBacked    493.0.0
com.apple.driver.usb.cdc    5.0.0
com.apple.driver.AppleUSBAudio  416.2
com.apple.iokit.IOAudioFamily   340.2
com.apple.vecLib.kext   1.2.0
com.apple.driver.AppleXsanScheme    3
com.apple.driver.AppleSEPHDCPManager    1.0.1
com.apple.driver.AppleTrustedAccessory  1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.AppleConvergedIPCOLYBTControl  1
com.apple.driver.AppleConvergedPCI  1
com.apple.driver.AppleBluetoothDebug    1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI    1
com.apple.driver.usb.AppleUSBXHCIPCI    1.2
com.apple.driver.AppleEmbeddedUSB   1
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.driver.AppleThunderboltDPInAdapter    8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily    8.5.1
com.apple.nke.ppp   1.9
com.apple.driver.AppleBSDKextStarter    3
com.apple.filesystems.hfs.encodings.kext    1
com.apple.driver.AppleBTM   1.0.1
com.apple.driver.AppleDiagnosticDataAccessReadOnly  1.0.0
com.apple.driver.AppleDCPDPTXProxy  1.0.0
com.apple.driver.DCPDPFamilyProxy   1
com.apple.driver.AppleI2CEthernetAquantia   1.0.0
com.apple.driver.AppleHPM   3.4.4
com.apple.driver.AppleCSEmbeddedAudio   550.3
com.apple.driver.ApplePassthroughPPM    3.0
com.apple.driver.usb.AppleUSBHostPacketFilter   1.0
com.apple.driver.AppleAOPAudio  150.2
com.apple.driver.AppleEmbeddedAudio 550.3
com.apple.iokit.AppleARMIISAudio    140.2
com.apple.AGXFirmwareKextG13XRTBuddy    190.22
com.apple.AGXFirmwareKextRTBuddy64  190.22
com.apple.driver.DCPAVFamilyProxy   1
com.apple.driver.AppleStockholmControl  1.0.0
com.apple.iokit.IONVMeFamily    2.1.0
com.apple.driver.AppleNANDConfigAccess  1.0.0
com.apple.driver.AppleMCA2-T600x    640.5
com.apple.driver.AppleEmbeddedAudioLibs 140.1
com.apple.driver.ApplePMGR  1
com.apple.driver.AppleSPU   1
com.apple.iokit.IOGPUFamily 35.29
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.driver.AppleUSBXDCIARM    1.0
com.apple.driver.AppleUSBXDCI   1.0
com.apple.iokit.IOUSBDeviceFamily   2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI   1
com.apple.driver.usb.AppleUSBXHCI   1.2
com.apple.driver.AppleEmbeddedUSBHost   1
com.apple.driver.usb.AppleUSBHub    1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice    1.2
com.apple.iokit.IOMobileGraphicsFamily-DCP  343.0.0
com.apple.driver.AppleDCP   1
com.apple.driver.AppleFirmwareKit   1
com.apple.driver.AppleFirmwareUpdateKext    1
com.apple.iokit.IOMobileGraphicsFamily  343.0.0
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleSPMI  1.0.1
com.apple.driver.AppleSART  1
com.apple.driver.AppleARMWatchdogTimer  1
com.apple.driver.AppleDisplayCrossbar   1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy  1
com.apple.driver.AppleThunderboltNHI    7.2.81
com.apple.driver.AppleT6000PCIeC    1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA    1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleMultiFunctionManager  1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore   1.0.0
com.apple.iokit.IO80211Family   1200.12.2b2
com.apple.driver.IOImageLoader  1.0.0
com.apple.driver.AppleOLYHAL    1
com.apple.driver.corecapture    1.0.4
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor 1.0.0
com.apple.driver.AppleEmbeddedPCIE  1
com.apple.driver.AppleGPIOICController  1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMobileApNonce 1
com.apple.iokit.IOTimeSyncFamily    1040.3
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily    597
com.apple.iokit.IOBluetoothSerialManager    9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport   9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport  9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport  9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily   9.0.0
com.apple.driver.FairPlayIOKit  68.16.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport  9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE   1.0
com.apple.driver.AppleSEPKeyStore   2
com.apple.driver.AppleUSBTDM    533.120.2
com.apple.iokit.IOUSBMassStorageDriver  210.120.3
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice   456.140.3
com.apple.iokit.IOSCSIArchitectureModelFamily   456.140.3
com.apple.driver.AppleIPAppender    1.0
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost    1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties    1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC   3.1.9
com.apple.driver.RTBuddy    1.0.0
com.apple.driver.AppleEmbeddedTempSensor    1.0.0
com.apple.driver.AppleARMPMU    1.0
com.apple.iokit.IOAccessoryManager  1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient  1.0.1b8
com.apple.iokit.IONetworkingFamily  3.4
com.apple.iokit.IOSerialFamily  11
com.apple.driver.AppleSEPManager    1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor   1
com.apple.driver.AppleBiometricSensor   2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.driver.AppleANELoadBalancer   5.68.2
com.apple.driver.AppleH11ANEInterface   5.68.0
com.apple.AUC   1.0
com.apple.iokit.IOAVFamily  1.0.0
com.apple.iokit.IOHDCPFamily    1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family  1.0
com.apple.driver.AppleIISController 140.1
com.apple.driver.AppleAudioClockLibs    140.1
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface   302.14
com.apple.driver.IODARTFamily   1
com.apple.security.quarantine   4
com.apple.security.sandbox  300.0
com.apple.kext.AppleMatch   1.0.0d1
com.apple.driver.AppleMobileFileIntegrity   1.0.5
com.apple.security.AppleImage4  4.2.0
com.apple.kext.CoreTrust    1
com.apple.iokit.IOCryptoAcceleratorFamily   1.0.1
com.apple.driver.AppleARMPlatform   1.0.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily    1.0.0
com.apple.iokit.IOReportFamily  47
com.apple.kec.pthread   1
com.apple.kec.Libm  1
com.apple.kec.corecrypto    12.0

** Stackshot Succeeded ** Bytes Traced 869802 (Uncompressed 2378112) **

[matthewwo]

lundman just released a new beta version Monterey for arm64, perhaps you can give it a shot? https://github.com/openzfsonosx/zfs/issues/798

[dmzimmerman]

The new version doesn't help.

[rottegift]

Hi, as of the start of this week I have one of these 128GiB M1 Ultras too, and now know what this problem is. Bear with us, this is the first time we've had a chance to run on a big arm Mac "in anger", and doing kernel extension development on the arm Macs is somewhat different in practice from doing it on Intel Macs.

I will commit a fix (and we will put out a package) in a couple of days, but in the meanwhile there is a workaround that should solve this for you entirely.

zsh$ sudo sysctl kstat.zfs.darwin.tunable.zfs_arc.max=$((16*1024**3))

You can also add the line

kstat.zfs.darwin.tunable.zfs_arc.max=17179869184

into /etc/zfs/zsysctl.conf. At boot time, by default, /usr/local/sbin/zsysctl will be run and will set the tunable for you.

This caps the maximum arc to 16 GiB. Almost certainly you can dial that up, but keep it well well well below 64 GIB, which is what the default settinig (0) uses. You can run the sysctl at any time the kernel extension is loaded and running, and dial it up and down; it should be safe enough to do so if you are sufficiently below half your system RAM. 16 GiB is very conservative. I have tested that value extensively and cannot get the machine to panic.

The problem in a nutshell is that we limit the entire zfs kernel extension (practically but not exactly all its allocations) to half of a system's RAM. In our case, 64 GIB. ARC is in present code capped by that 64 GiB, rather than some lower value. This leads to two potential problems.

The first problem is that because of the constraints on knowing how much memory is available from the kernel to our kernel extension, we are reactive to memory pressure. It's hard to cause a 128 GIB mac to run critically low on memory, so on my system so far I have been unable to see a pressure event at all (and I have tried!). As a result ARC may grow to its maximum extent, 64 GiB, because nothing is telling it to stop growing. Unfortunately this leaves no room for the other parts of our kernel extension which need memory.

The second problem arises when it's discovered by our code that nearly all 64GiB are in use. When that's observed some low-level code generates an artificial pressure event, and waits for that to take effect. The signal was not strong (a multiple of the desired allocation size, which translates into between kilobytes and a couple megabytes). On a machine like the M1 Studios, with their many fast I/O ports and little chance of being CPU bound when doing I/O, demands for new memory can easily outrace these attempts to free small amounts, leading to a period of allocations stalling.

The result is that a client I/O (say, HFS or APFS inside a zvol; or our command line zfs and zpool tools, or running /bin/ls on a dataset) can be delayed for potentially a long time under certain system workloads, and some of those clients are intolerant of such delays. In particular, HFS appears to decide to time out and panic in some cases.

Two probable fixes for this are in the code I am running and testing now, and will in due course commit to our trees. One change scales the artificial pressure signal up with increasing system memory. The other caps the ARC to below the 50% total extension memory threshold.

This will get rid of this exact panic, which I can reproduce easily (keep the arc_max tunable at the default, and do a time machine backup into a zvol on an otherwise quiet system). It will also avoid other panics that can arise from the same root cause.

[dmzimmerman]

Awesome! I will try that right now and see what happens; do let me know if I can otherwise help with testing in any way.

Now if we can just figure out what causes the boot loop when installing the same kext in Ventura... :)

[dmzimmerman]

I can use sysctl to set the tunable manually, but apparently /usr/local/zfs/bin/zsysctl is not running automatically on reboot, because on reboot the tunable is back to 0 (and the machine panics).

[jawbroken]

Great debugging work @rottegift, I really appreciate it.

[rottegift]

@dmzimmerman do a "sudo launchctl list | grep -i zfs", and see if it says something like:

759 1   org.openzfsonosx.zed
143 0   org.openzfsonosx.zconfigd
145 0   org.openzfsonosx.InvariantDisks

The columns are process ID, status, and launchctl job label.

also, "sudo launchctl list org.openzfsonosx.zconfigd"

{
    "StandardOutPath" = "/private/var/log/org.openzfsonosx.zconfigd.log";
    "LimitLoadToSessionType" = "System";
    "StandardErrorPath" = "/private/var/log/org.openzfsonosx.zconfigd.err";
    "Label" = "org.openzfsonosx.zconfigd";
    "OnDemand" = false;
    "LastExitStatus" = 0;
    "PID" = 143;
    "Program" = "/usr/local/zfs/sbin/zconfigd";
    "ProgramArguments" = (
        "/usr/local/zfs/sbin/zconfigd";
    );
};

And do look in /var/log/org.openzfsonosx.log and /var/log/org.openzfsonosx.err to see if there's any evidence of failures (or even zconfigd starting).

zconfigd should be running

/usr/local/zfs/sbin/zsysctl -f /etc/zfs/zsysctl.conf

You could run that by hand (use sudo) to see that it emits any output (it should say at least something like:

kstat.zfs.darwin.tunable.zfs_arc.max: 0 -> 17179869184

@jawbroken : thank you. I'm still working on a few ARM things I discovered, and will land it all at once in a few days. As far as I can tell on a 128GiB machine it is safe to use 16 GiB arc_max. With modern compressed ARC that hopefully that's not such a low constraint to suffer with for a few days.

[dmzimmerman]

Unfortunately, I have to report that it still panics for me, even with a 16GiB arc_max. It takes a lot longer, though, and I was pushing it pretty hard.

panic(cpu 2 caller 0xfffffe0015831f1c): Kernel data abort. at pc 0xfffffe0016bdadb4, lr 0xfffffe0016bdad9c (saved state: 0xfffffe180b38f3c0)
      x0:  0x0000000000000000 x1:  0x0000000000300000  x2:  0x0000000000000000  x3:  0xfffffe1810cc6ca0
      x4:  0x0000000000000000 x5:  0x000001477e03ffee  x6:  0x0000000000000000  x7:  0xfffffe180b38f568
      x8:  0x0000000000000000 x9:  0xb5c52280151d0028  x10: 0x0000000000000000  x11: 0x0000000000000000
      x12: 0x0000000000000001 x13: 0x0000000000000014  x14: 0x0000000000153be4  x15: 0x00000000ffffffff
      x16: 0xf486fe00152a1a2c x17: 0xfffffe0018294a58  x18: 0x0000000000000000  x19: 0xfffffe2cdd6f0200
      x20: 0xfffffe2344933300 x21: 0x0000000000000000  x22: 0x0000000000300000  x23: 0x0000000000000000
      x24: 0x0000000000000000 x25: 0xfffffe180b38f820  x26: 0xfffffe0018b35920  x27: 0x0000000000000000
      x28: 0x0000000003018001 fp:  0xfffffe180b38f740  lr:  0xfffffe0016bdad9c  sp:  0xfffffe180b38f710
      pc:  0xfffffe0016bdadb4 cpsr: 0x60401208         esr: 0x96000046          far: 0x0000000000000000

Debugger message: panic
Memory ID: 0x6
OS release type: User
OS version: 21G115
Kernel version: Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000
Fileset Kernelcache UUID: 39A7E336B0FAA0022B3764E49DFF29D2
Kernel UUID: 778CC57A-CF0B-3D35-8EE8-5035142D0177
iBoot version: iBoot-7459.141.1
secure boot?: YES
Paniclog version: 13
KernelCache slide: 0x000000000d6f8000
KernelCache base:  0xfffffe00146fc000
Kernel slide:      0x000000000debc000
Kernel text base:  0xfffffe0014ec0000
Kernel text exec slide: 0x000000000dfa4000
Kernel text exec base:  0xfffffe0014fa8000
mach_absolute_time: 0x1477e12d85a
Epoch Time:        sec       usec
  Boot    : 0x632ab6f2 0x000e9f90
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x632b9bd8 0x0008165c

Zone info:
  Zone map: 0xfffffe1810ca8000 - 0xfffffe3810ca8000
  . VM    : 0xfffffe1810ca8000 - 0xfffffe1cdd974000
  . RO    : 0xfffffe1cdd974000 - 0xfffffe1e7730c000
  . GEN0  : 0xfffffe1e7730c000 - 0xfffffe2343fd8000
  . GEN1  : 0xfffffe2343fd8000 - 0xfffffe2810ca4000
  . GEN2  : 0xfffffe2810ca4000 - 0xfffffe2cdd970000
  . GEN3  : 0xfffffe2cdd970000 - 0xfffffe31aa63c000
  . DATA  : 0xfffffe31aa63c000 - 0xfffffe3810ca8000
  Metadata: 0xfffffe3810cb8000 - 0xfffffe3818cb8000
  Bitmaps : 0xfffffe3818cb8000 - 0xfffffe3838cb8000

CORE 0 PVH locks held: None
CORE 1 PVH locks held: None
CORE 2 PVH locks held: None
CORE 3 PVH locks held: None
CORE 4 PVH locks held: None
CORE 5 PVH locks held: None
CORE 6 PVH locks held: None
CORE 7 PVH locks held: None
CORE 8 PVH locks held: None
CORE 9 PVH locks held: None
CORE 10 PVH locks held: None
CORE 11 PVH locks held: None
CORE 12 PVH locks held: None
CORE 13 PVH locks held: None
CORE 14 PVH locks held: None
CORE 15 PVH locks held: None
CORE 16 PVH locks held: None
CORE 17 PVH locks held: None
CORE 18 PVH locks held: None
CORE 19 PVH locks held: None
CORE 0: PC=0xfffffe001505e144, LR=0xfffffe001505e144, FP=0xfffffe180b54fd40
CORE 1: PC=0xfffffe001501c8a4, LR=0xfffffe00154f75c4, FP=0xfffffe8fedb0ff20
CORE 2 is the one that panicked. Check the full backtrace for details.
CORE 3: PC=0xfffffe00150cf504, LR=0xfffffe00150cde6c, FP=0xfffffe180ac2f950
CORE 4: PC=0xfffffe0015037258, LR=0xfffffe0015037254, FP=0xfffffe8fec44ff00
CORE 5: PC=0xfffffe001513bfb4, LR=0xfffffe001513bfb0, FP=0xfffffe8fed7efe90
CORE 6: PC=0xfffffe0015037258, LR=0xfffffe0015037254, FP=0xfffffe180f16ff00
CORE 7: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180e44ff00
CORE 8: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe88069cff00
CORE 9: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe8fee52ff00
CORE 10: PC=0x00000001afe9abc8, LR=0x00000001afe9ab94, FP=0x000000016f3da770
CORE 11: PC=0xfffffe001501c588, LR=0xfffffe001501c910, FP=0xfffffe880700fd10
CORE 12: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180f32ff00
CORE 13: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180f0eff00
CORE 14: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180b64ff00
CORE 15: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180e00ff00
CORE 16: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180a78ff00
CORE 17: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180b5cff00
CORE 18: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180b42ff00
CORE 19: PC=0xfffffe0015037254, LR=0xfffffe0015037254, FP=0xfffffe180e38ff00
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 1 swapfiles and OK swap space
Panicked task 0xfffffe2810df66d0: 2310 pages, 4 threads: pid 1: launchd
Panicked thread: 0xfffffe31a7134820, backtrace: 0xfffffe180b38ea80, tid: 989660
          lr: 0xfffffe0015001400  fp: 0xfffffe180b38eaf0
          lr: 0xfffffe00150010c8  fp: 0xfffffe180b38eb60
          lr: 0xfffffe001514733c  fp: 0xfffffe180b38eb80
          lr: 0xfffffe00151390cc  fp: 0xfffffe180b38ebf0
          lr: 0xfffffe0015136cb0  fp: 0xfffffe180b38ecb0
          lr: 0xfffffe0014faf7f8  fp: 0xfffffe180b38ecc0
          lr: 0xfffffe0015000d4c  fp: 0xfffffe180b38f060
          lr: 0xfffffe0015000d4c  fp: 0xfffffe180b38f0d0
          lr: 0xfffffe001582954c  fp: 0xfffffe180b38f0f0
          lr: 0xfffffe0015831f1c  fp: 0xfffffe180b38f270
          lr: 0xfffffe0015138ecc  fp: 0xfffffe180b38f2e0
          lr: 0xfffffe0015136fb4  fp: 0xfffffe180b38f3a0
          lr: 0xfffffe0014faf7f8  fp: 0xfffffe180b38f3b0
          lr: 0xfffffe0016bdad9c  fp: 0xfffffe180b38f740
          lr: 0xfffffe0016c0ac20  fp: 0xfffffe180b38f7a0
          lr: 0xfffffe0016bca03c  fp: 0xfffffe180b38f9a0
          lr: 0xfffffe0016bcdc78  fp: 0xfffffe180b38fa10
          lr: 0xfffffe0016bf8d14  fp: 0xfffffe180b38fa40
          lr: 0xfffffe0015268b24  fp: 0xfffffe180b38fad0
          lr: 0xfffffe0016bf5c4c  fp: 0xfffffe180b38fce0
          lr: 0xfffffe001527c864  fp: 0xfffffe180b38fd20
          lr: 0xfffffe00152688c8  fp: 0xfffffe180b38fda0
          lr: 0xfffffe001527c7ac  fp: 0xfffffe180b38fdb0
          lr: 0xfffffe001566aa28  fp: 0xfffffe180b38fe50
          lr: 0xfffffe0015136d84  fp: 0xfffffe180b38ff10
          lr: 0xfffffe0014faf7f8  fp: 0xfffffe180b38ff20
      Kernel Extensions in backtrace:
         com.apple.filesystems.hfs.kext(583.100.10)[45F25204-8A60-3A88-B71F-974BDDBDB3BF]@0xfffffe0016bc48a0->0xfffffe0016c134e3
            dependency: com.apple.filesystems.hfs.encodings.kext(1)[4183166A-286A-3CEB-8C2C-AF85AA1F4D16]@0xfffffe0016c134f0->0xfffffe0016c1441f

last started kext at 3195486023: com.apple.filesystems.smbfs    4.0 (addr 0xfffffe0014ea4c30, size 65195)
loaded kexts:
org.openzfsonosx.zfs    2.1.99
com.apple.filesystems.smbfs 4.0
com.apple.filesystems.autofs    3.0
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL    1
com.apple.AppleEthernetAquantiaAqtionFirmware   1.0.36
com.apple.driver.AppleAHCIPort  351.100.4
com.apple.driver.DiskImages.ReadWriteDiskImage  493.0.0
com.apple.driver.DiskImages.UDIFDiskImage   493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore    493.0.0
com.apple.driver.BCMWLANFirmware4387.Hashstore  1
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleUSBDeviceNCM  5.0.0
com.apple.driver.AppleFileSystemDriver  3.0.1
com.apple.nke.l2tp  1.9
com.apple.filesystems.tmpfs 1
com.apple.driver.ApplePMP   1
com.apple.driver.AppleSmartIO2  1
com.apple.filesystems.lifs  1
com.apple.filesystems.apfs  1934.141.2
com.apple.IOTextEncryptionFamily    1.0.0
com.apple.filesystems.hfs.kext  583.100.10
com.apple.security.BootPolicy   1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleEmbeddedSimpleSPINORFlasher  1
com.apple.driver.AppleSN012776Amp   550.3
com.apple.driver.AppleCS42L84Audio  550.3
com.apple.driver.AppleT6000SOCTuner 1
com.apple.driver.AppleT6002CLPC 1
com.apple.driver.AppleSmartBatteryManager   161.0.0
com.apple.driver.AppleAOPVoiceTrigger   140.1
com.apple.driver.ApplePMPFirmware   1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AppleSerialShim    1
com.apple.driver.AppleSDXC  3.2.1
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleMCDP29XXUpdateSupport 1
com.apple.driver.AppleS5L8960XNCO   1
com.apple.driver.AudioDMAController-T600x   160.2
com.apple.driver.AppleSPMIPMU   1.0.1
com.apple.driver.AppleAVE2  560.5.0
com.apple.driver.AppleAVD   566
com.apple.driver.AppleS8000AES  1
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleEventLogHandler   1
com.apple.AGXG13X   190.22
com.apple.driver.AppleS5L8920XPWM   1.0.0d1
com.apple.driver.AppleT6001PMGR 1
com.apple.driver.AppleJPEGDriver    4.8.1
com.apple.driver.AppleProResHW  177.0.0
com.apple.driver.AppleMobileDispT600X-DCP   140.0
com.apple.driver.AppleS8000DWI  1.0.0d1
com.apple.driver.AppleInterruptControllerV2 1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleBluetoothModule   1
com.apple.driver.AppleBCMWLANBusInterfacePCIe   1
com.apple.driver.AppleS5L8940XI2C   1.0.0d2
com.apple.driver.AppleT6000DART 1
com.apple.driver.AppleT6000 1
com.apple.driver.AppleM68Buttons    1.0.0d1
com.apple.iokit.IOUserEthernet  1.0.1
com.apple.driver.usb.AppleUSBUserHCI    1
com.apple.iokit.IOKitRegistryCompatibility  1
com.apple.iokit.EndpointSecurity    1
com.apple.driver.AppleDiskImages2   126.141.2
com.apple.AppleSystemPolicy 2.0.0
com.apple.nke.applicationfirewall   402
com.apple.kec.InvalidateHmac    1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial   6.0.0
com.apple.iokit.IOAVBFamily 1040.6
com.apple.plugin.IOgPTPPlugin   1040.3
com.apple.iokit.IOEthernetAVBController 1.1.0
com.apple.iokit.IOAHCIBlockStorage  333.140.2
com.apple.kext.triggers 1.0
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily   1
com.apple.iokit.IOAHCIFamily    297
com.apple.driver.AppleEthernetAquantiaAqtion    1.0.64
com.apple.driver.AppleThunderboltPCIUpAdapter   4.1.1
com.apple.driver.AppleThunderboltDPOutAdapter   8.5.1
com.apple.driver.DiskImages.KernelBacked    493.0.0
com.apple.driver.AppleXsanScheme    3
com.apple.driver.AppleSEPHDCPManager    1.0.1
com.apple.driver.AppleTrustedAccessory  1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.usb.AppleEmbeddedUSBXHCIPCI    1
com.apple.driver.usb.AppleUSBXHCIPCI    1.2
com.apple.driver.AppleEmbeddedUSB   1
com.apple.driver.AppleConvergedIPCOLYBTControl  1
com.apple.driver.AppleConvergedPCI  1
com.apple.driver.AppleBluetoothDebug    1
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.driver.AppleThunderboltDPInAdapter    8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily    8.5.1
com.apple.driver.AppleBTM   1.0.1
com.apple.nke.ppp   1.9
com.apple.driver.AppleDCPDPTXProxy  1.0.0
com.apple.driver.DCPDPFamilyProxy   1
com.apple.driver.AppleBSDKextStarter    3
com.apple.filesystems.hfs.encodings.kext    1
com.apple.driver.AppleDiagnosticDataAccessReadOnly  1.0.0
com.apple.driver.AppleCSEmbeddedAudio   550.3
com.apple.driver.AppleHPM   3.4.4
com.apple.driver.AppleI2CEthernetAquantia   1.0.0
com.apple.driver.ApplePassthroughPPM    3.0
com.apple.driver.AppleAOPAudio  150.2
com.apple.driver.AppleEmbeddedAudio 550.3
com.apple.iokit.AppleARMIISAudio    140.2
com.apple.iokit.IONVMeFamily    2.1.0
com.apple.driver.AppleNANDConfigAccess  1.0.0
com.apple.driver.AppleSART  1
com.apple.driver.AppleSPU   1
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleStockholmControl  1.0.0
com.apple.driver.AppleSPMI  1.0.1
com.apple.driver.DCPAVFamilyProxy   1
com.apple.driver.AppleMCA2-T600x    640.5
com.apple.driver.AppleEmbeddedAudioLibs 140.1
com.apple.driver.AppleUSBXDCIARM    1.0
com.apple.driver.AppleUSBXDCI   1.0
com.apple.iokit.IOUSBDeviceFamily   2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI   1
com.apple.driver.usb.AppleUSBXHCI   1.2
com.apple.driver.AppleEmbeddedUSBHost   1
com.apple.driver.usb.AppleUSBHub    1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice    1.2
com.apple.driver.AppleT6000TypeCPhy 1
com.apple.driver.AppleT8103TypeCPhy 1
com.apple.AGXFirmwareKextG13XRTBuddy    190.22
com.apple.AGXFirmwareKextRTBuddy64  190.22
com.apple.iokit.IOGPUFamily 35.29
com.apple.driver.ApplePMGR  1
com.apple.driver.usb.AppleUSBHostPacketFilter   1.0
com.apple.iokit.IOMobileGraphicsFamily-DCP  343.0.0
com.apple.driver.AppleDCP   1
com.apple.driver.AppleFirmwareKit   1
com.apple.driver.AppleFirmwareUpdateKext    1
com.apple.iokit.IOMobileGraphicsFamily  343.0.0
com.apple.driver.AppleARMWatchdogTimer  1
com.apple.driver.AppleDisplayCrossbar   1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleTypeCPhy  1
com.apple.driver.AppleThunderboltNHI    7.2.81
com.apple.driver.AppleT6000PCIeC    1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.driver.ApplePIODMA    1
com.apple.driver.AppleT600xPCIe 1
com.apple.driver.AppleMultiFunctionManager  1
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore   1.0.0
com.apple.iokit.IO80211Family   1200.12.2b2
com.apple.driver.IOImageLoader  1.0.0
com.apple.driver.AppleOLYHAL    1
com.apple.driver.corecapture    1.0.4
com.apple.driver.AppleEthernetAquantiaAqtionPortMonitor 1.0.0
com.apple.driver.AppleEmbeddedPCIE  1
com.apple.driver.AppleGPIOICController  1.0.2
com.apple.driver.AppleFireStormErrorHandler 1
com.apple.driver.AppleMobileApNonce 1
com.apple.iokit.IOTimeSyncFamily    1040.3
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily    597
com.apple.iokit.IOBluetoothSerialManager    9.0.0
com.apple.iokit.IOBluetoothHostControllerUSBTransport   9.0.0
com.apple.iokit.IOBluetoothHostControllerUARTTransport  9.0.0
com.apple.iokit.IOBluetoothHostControllerTransport  9.0.0
com.apple.driver.IOBluetoothHostControllerPCIeTransport 9.0.0
com.apple.iokit.IOBluetoothFamily   9.0.0
com.apple.driver.FairPlayIOKit  68.16.0
com.apple.iokit.CSRBluetoothHostControllerUSBTransport  9.0.0
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 9.0.0
com.apple.driver.AppleSSE   1.0
com.apple.driver.AppleSEPKeyStore   2
com.apple.driver.AppleUSBTDM    533.120.2
com.apple.iokit.IOUSBMassStorageDriver  210.120.3
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOSCSIBlockCommandsDevice   456.140.3
com.apple.iokit.IOSCSIArchitectureModelFamily   456.140.3
com.apple.driver.AppleIPAppender    1.0
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.KernelRelayHost    1
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties    1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC   3.1.9
com.apple.driver.RTBuddy    1.0.0
com.apple.driver.AppleEmbeddedTempSensor    1.0.0
com.apple.driver.AppleARMPMU    1.0
com.apple.iokit.IOAccessoryManager  1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient  1.0.1b8
com.apple.iokit.IONetworkingFamily  3.4
com.apple.iokit.IOSerialFamily  11
com.apple.driver.AppleSEPManager    1.0.1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor   1
com.apple.driver.AppleBiometricSensor   2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.driver.AppleANELoadBalancer   5.68.2
com.apple.driver.AppleH11ANEInterface   5.68.0
com.apple.AUC   1.0
com.apple.iokit.IOAVFamily  1.0.0
com.apple.iokit.IOHDCPFamily    1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family  1.0
com.apple.driver.AppleIISController 140.1
com.apple.driver.AppleAudioClockLibs    140.1
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.iokit.IOSurface   302.14
com.apple.driver.IODARTFamily   1
com.apple.security.quarantine   4
com.apple.security.sandbox  300.0
com.apple.kext.AppleMatch   1.0.0d1
com.apple.driver.AppleMobileFileIntegrity   1.0.5
com.apple.security.AppleImage4  4.2.0
com.apple.kext.CoreTrust    1
com.apple.iokit.IOCryptoAcceleratorFamily   1.0.1
com.apple.driver.AppleARMPlatform   1.0.2
com.apple.iokit.IOStorageFamily 2.1
com.apple.iokit.IOSlowAdaptiveClockingFamily    1.0.0
com.apple.iokit.IOReportFamily  47
com.apple.kec.pthread   1
com.apple.kec.Libm  1
com.apple.kec.corecrypto    12.0

** Stackshot Succeeded ** Bytes Traced 762271 (Uncompressed 2090224) **

[dmzimmerman]

zconfigd should be running

/usr/local/zfs/sbin/zsysctl -f /etc/zfs/zsysctl.conf

Well, I think I found the problem:

• first, zsysctl is in /usr/local/zfs/bin on my machine, and there is no /usr/local/zfs/sbin.
• but, assuming that's a typo, when I run sudo /usr/local/zfs/bin/zsysctl -f /etc/zfs/zsysctl.conf, it does kstat.zfs.darwin.tunable.zfs_arc.max: 17179869184 -> 0, which is not at all what I expected to see.

The contents of my /etc/zfs/zsysctl.conf are:

kstat.zfs.darwin.tunable.zfs_arc.max=17179869184

so I really don't understand what's happening there.

[rottegift]

Update:

tl;dr: kernel doesn't like giving our kext more than approx 32 GIB of memory on 128GiB Mac Studios.

Previously in our second lowest layer of memory allocation there was plumbing to wait in a loop for memory to appear in some circumstances. On our systems, those circumstances were always us running close to 50% of system RAM (a self-imposed limit) or learning from the kernel that system memory was recently (or even currently) in short supply.

The waiting would also happen if we received a NULL from the (essentially) malloc "system" call to the kernel.

For years we were using a since-deprecated kernel malloc call that was guaranteed never to fail: it would give you the memory requested or panic the system. More recently we switched to IOMallocAligned(), which it turns out is allowed to return a NULL whenever it likes.

As far as I can tell, we have only ever seen many NULLs being returned from IOMallocAligned() on these 128GiB Mac Studios. (I guess it could have happenend on an Intel Mac or Hack with 128GiB+ system RAM, but a brief search showed no complaints suggesting anyone had discovered this bug; it could be that on Intel platforms IOMallocAligned() returns NULL so rarely that it might as well be never -- one might never notice the effect of a NULL if retrying produced a successful allocation within a few milliseconds or a few thousand passes through the waiting loop).

I ran an experiment to simplify our second-lowest allocation layer on macOS 12+ builds. I think the 12+ condition is reasonable -- macOS Monterrey benefits from years of memory system development by Apple, and additionally all our Mac Studio 128GiB machines, when running macOS, must be running at 12+.

The results are promising. As expected, ARC grows as quickly as it can, causing (directly and indirectly) us to allocate a little more (!) than 32 GiB of memory, after which we start seeing NULL almost every time we use IOMallocAligned() for more. Nothing seems to break, and ARC correctly transitions into a much more slowly-and-tentatively-growing mode, causing us to use IOMallocAligned() much less frequently.

I can think of a few ways to try to get past the (slightly-more-than-)32GiB effective limit already, but want to UTSL and /usr/bin/zprint to see how exactly it's enforced on us.

However, give or take cosmetic changes, I have a workaround that avoids the panic without the need to set an explicit non-default arc_max at runtime via sysctl. I am likely to commit the change either tonight or Monday.

[ETA (pardon the acronym pun), more likely Monday or Tuesday, as I will want to test whether a variety of other Macs and VMs, with different OS vintages, are happy with a total removal of the relevant code, instead of an #ifdef removing it for macOS 12+ only. Over time the amount of time we have to pester xnu with requests for more memory or to return memory have reduced via changes in middle layers of our memory management. However I don't want to make a change that causes panics on 10.11 or even earlier, or on machines (including virtual machines) with only 4 GiB total RAM].

Until then, setting a sufficiently low arc_max via sysctl will virtually certainly prevent the entire kext from attempting to allocate more than 32 GiB memory, and thus will not have to deal with these strings of NULLs from IOMallocAligned().

16GiB is still my best advice for the upper limit. With this experimental code I am seeing a 26GiB ARC at the moment, and I have seen at least as much as kstat.spl.misc.spl_misc.os_mem_alloc: 35,670,573,056 memory having been obtained from osif_malloc()->IOMallocAligned(). So my kext is using somewhat, but not much, more than 33 GiB (we make some allocations via other kernel interfaces, too, and those don't pass through osif_malloc() and therefore don't increment that particular kstat).

@dmzimmerman : the only thing I can think of is that you may have installed a recent package built by lundman without first making sure that a previous install was truly gone. Personally, i prefer /usr/local/zfs/{bin,sbin,libexec,...} to /usr/local/bin et al. I did get surprised in resorting to a pacakge lundman built for me personally for bootstrapping purposes, because I had always built my own installation (rather than using a release) and had put zpool, zfs etc. in /usr/local/sbin, and so had to update the PATHs of a few scripts of mine. However I had cleaned out (post-migration-from-intel-mac) my previously installed openzfsonosx code, so zconfigd/zsysctl was doing the right thing.

I think (having done a very brief UTSL) that at least since May 2021 packages have been using /usr/local/zfs.

[dmzimmerman]

@rottegift: I think you may have misread my comment (or I don't understand what you're saying in response). My zsysctl is in /usr/local/zfs/bin, not /usr/local/zfs/sbin (as you expected), and definitely not in /usr/local/sbin or /usr/local/bin (which have no ZFS binaries); but the problem isn't that it doesn't exist or is in the wrong place, the problem is that zsysctl resets the tunable to 0 instead of actually doing what I have set in the configuration file, when I run it by hand. (And the other problem, of course, is that I'm still seeing the same panics even when setting the low arc_max by hand; should I go lower than 16GB?)

[rottegift]

@dmzimmerman - The output of zsysctl should be identical to if you copied and pasted the non-commented-out lines from /etc/zfs/zsysctl.conf to just after "sudo sysctl ". What you are reporting is unexpected, and not what I have ever seen from zsysctl. It's also not a complex tool, in coding terms.

I don't know the provenance of your zsysctl. Packaging has never been my department, and it might be that the whatever installed your zsysctl correctly put it there (and likewise put all the other userland files in concordant locations). My guess was that your zsysctl is old (and maybe file and otool -L on it can prove this), but that's only a guess. Is there any chance your zsysctl is being run under rosetta? This is really a question for lundman, and frankly the subject for a different issue number / standalone problem report. (FWIW, he did scratch his head about it yesterday on IRC).

Yes, you should try going lower than 16GiB. Off the top of my head, I can think of workloads that might drive the ARC-size-to-total-memory-allocated ratio upwards. However, do verify that the sysctl has been set, in case there is any doubt about that. If you it is set before import or other activity might exercise ARC, you should see e.g.:

% sudo sysctl kstat.zfs.darwin.tunable.zfs_arc.max=429467296
kstat.zfs.darwin.tunable.zfs_arc.max: 0 -> 429467296
% sysctl -a | grep 429467296
kstat.zfs.darwin.tunable.zfs_arc.max: 429467296
kstat.zfs.misc.arcstats.c_max 429467296

(The last line reports a variable generated by ARC).

[dmzimmerman]

Fair enough. My zsysctl dates from September 8 (in the most recent package lundman built), and is definitely a universal binary. And I did verify that the crashes occurred with the sysctl actually set. I've lowered it to 8589934592 (8 GB) now and we'll see if that does any better. And I'll file an issue about zsysctl.