Panic: "zalloc: zone map exhausted while allocating from zone kalloc.64"

[codutti]

Today and Yesterday I've received two panics. The second one produced this report:

Wed Mar 26 23:23:19 2014
panic(cpu 2 caller 0xffffff802fc5220d): "zalloc: zone map exhausted while allocating from zone kalloc.64, likely due to memory leak in zone kalloc.256 (3888533504 total bytes, 15163226 elements allocated)"@/SourceCache/xnu/xnu-2422.90.20/osfmk/kern/zalloc.c:2494
Backtrace (CPU 2), Frame : Return Address
0xffffff8231c13b50 : 0xffffff802fc22fa9 
0xffffff8231c13bd0 : 0xffffff802fc5220d 
0xffffff8231c13cd0 : 0xffffff802fc2aa6f 
0xffffff8231c13d00 : 0xffffff802fc2b179 
0xffffff8231c13d30 : 0xffffff7fb0218156 
0xffffff8231c13d60 : 0xffffff7fb02b99f3 
0xffffff8231c13dc0 : 0xffffff7fb02b8e9d 
0xffffff8231c13e80 : 0xffffff7fb0320cdc 
0xffffff8231c13ef0 : 0xffffff7fb031dbc8 
0xffffff8231c13f30 : 0xffffff7fb031da95 
0xffffff8231c13f50 : 0xffffff7fb021a63d 
0xffffff8231c13fb0 : 0xffffff802fcd6ff7 
      Kernel Extensions in backtrace:
         net.lundman.spl(1.0)[F40D547D-33B9-3E7E-92DD-B020B0A29F96]@0xffffff7fb0217000->0xffffff7fb0223fff
         net.lundman.zfs(1.0)[895CD7C5-59E2-354D-89D8-F987FCFEEA28]@0xffffff7fb0224000->0xffffff7fb0426fff
            dependency: com.apple.iokit.IOStorageFamily(1.9)[9B09B065-7F11-3241-B194-B72E5C23548B]@0xffffff7fb01e9000
            dependency: net.lundman.spl(1.0.0)[F40D547D-33B9-3E7E-92DD-B020B0A29F96]@0xffffff7fb0217000

BSD process name corresponding to current thread: kernel_task

Mac OS version:
13C64

Kernel version:
Darwin Kernel Version 13.1.0: Thu Jan 16 19:40:37 PST 2014; root:xnu-2422.90.20~2/RELEASE_X86_64
Kernel UUID: 9FEA8EDC-B629-3ED2-A1A3-6521A1885953
Kernel slide:     0x000000002fa00000
Kernel text base: 0xffffff802fc00000
System model name: Macmini5,2 (Mac-4BC72D62AD45599E)

System uptime in nanoseconds: 40049847417160
vm objects:55545952
vm object hash entri:8936480
VM map entries:7171280
Reserved VM map entr:4084240
pv_list:7372800
vm pages:295638984
kalloc.16:244013440
kalloc.32:477082368
kalloc.64:976981824
kalloc.128:15769600
kalloc.256:3888533504
kalloc.512:2588672
kalloc.1024:72843264
kalloc.4096:363851776
kalloc.8192:7446528
mem_obj_control:3582480
ipc ports:1513600
threads:1139328
vnodes:63162480
namecache:27727680
HFS node:86212832
HFS fork:30183424
cluster_read:4718720
cluster_write:5699288
ubc_info zone:20724352
vnode pager structur:8936480
Kernel Stacks:6193152
PageTables:50630656
Kalloc.Large:127017663

Backtrace suspected of leaking: (outstanding bytes: 266240)
0xffffff802fc51d13
0xffffff802fc2aa6f
0xffffff802fc2b179
0xffffff7fb021840f
0xffffff7fb031b948
0xffffff7fb0225ef6
0xffffff7fb0225bc3
0xffffff7fb023623a
0xffffff7fb0236101
0xffffff7fb02417a4
0xffffff7fb024161c
0xffffff7fb0329ee5
0xffffff7fb032c095
0xffffff7fb01ed720
0xffffff7fb01efcef
      Kernel Extensions in backtrace:
         com.apple.iokit.IOStorageFamily(1.9)[9B09B065-7F11-3241-B194-B72E5C23548B]@0xffffff7fb01e9000->0xffffff7fb020dfff
         net.lundman.spl(1.0)[F40D547D-33B9-3E7E-92DD-B020B0A29F96]@0xffffff7fb0217000->0xffffff7fb0223fff
         net.lundman.zfs(1.0)[895CD7C5-59E2-354D-89D8-F987FCFEEA28]@0xffffff7fb0224000->0xffffff7fb0426fff
            dependency: com.apple.iokit.IOStorageFamily(1.9)[9B09B065-7F11-3241-B194-B72E5C23548B]@0xffffff7fb01e9000
            dependency: net.lundman.spl(1.0.0)[F40D547D-33B9-3E7E-92DD-B020B0A29F96]@0xffffff7fb0217000

last loaded kext at 39688125084808: com.apple.driver.AppleIntelMCEReporter  104 (addr 0xffffff7fb25a4000, size 49152)
last unloaded kext at 39758977230491: com.apple.driver.AppleIntelMCEReporter    104 (addr 0xffffff7fb25a4000, size 32768)
loaded kexts:
net.lundman.zfs 1.0.0
net.lundman.spl 1.0.0
com.apple.driver.AudioAUUC  1.60
com.apple.driver.AppleHWSensor  1.9.5d0
com.apple.filesystems.autofs    3.0
com.apple.iokit.IOBluetoothSerialManager    4.2.3f10
com.apple.driver.AGPM   100.14.15
com.apple.driver.ApplePlatformEnabler   2.0.9d1
com.apple.driver.AppleMikeyHIDDriver    124
com.apple.driver.AppleMikeyDriver   2.6.0f1
com.apple.driver.AppleHDA   2.6.0f1
com.apple.iokit.BroadcomBluetoothHostControllerUSBTransport 4.2.3f10
com.apple.driver.ACPI_SMC_PlatformPlugin    1.0.0
com.apple.driver.AppleUpstreamUserClient    3.5.13
com.apple.driver.AppleMCCSControl   1.1.12
com.apple.driver.AppleLPC   1.7.0
com.apple.kext.AMDFramebuffer   1.2.0
com.apple.driver.AppleSMCPDRC   1.0.0
com.apple.driver.AppleThunderboltIP 1.1.2
com.apple.iokit.IOUserEthernet  1.0.0d1
com.apple.Dont_Steal_Mac_OS_X   7.0.0
com.apple.driver.AppleHWAccess  1
com.apple.AMDRadeonX3000    1.2.0
com.apple.kext.AMD6000Controller    1.2.0
com.apple.driver.AppleIntelHD3000Graphics   8.2.4
com.apple.driver.AppleIntelSNBGraphicsFB    8.2.4
com.apple.driver.AppleIRController  325.7
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0d1
com.apple.BootCache 35
com.apple.driver.XsanFilter 404
com.apple.iokit.IOAHCIBlockStorage  2.5.1
com.apple.driver.AppleUSBHub    666.4.0
com.apple.driver.AppleFWOHCI    4.9.9
com.apple.driver.AppleAHCIPort  3.0.0
com.apple.driver.AppleSDXC  1.5.2
com.apple.driver.AirPort.Brcm4331   700.20.22
com.apple.iokit.AppleBCM5701Ethernet    3.8.1b2
com.apple.driver.AppleUSBEHCI   660.4.0
com.apple.driver.AppleACPIButtons   2.0
com.apple.driver.AppleRTC   2.0
com.apple.driver.AppleHPET  1.8
com.apple.driver.AppleSMBIOS    2.1
com.apple.driver.AppleACPIEC    2.0
com.apple.driver.AppleAPIC  1.7
com.apple.driver.AppleIntelCPUPowerManagementClient 216.0.0
com.apple.nke.applicationfirewall   153
com.apple.security.quarantine   3
com.apple.driver.AppleIntelCPUPowerManagement   216.0.0
com.apple.kext.triggers 1.0
com.apple.iokit.IOSerialFamily  10.0.7
com.apple.iokit.IOSCSIArchitectureModelFamily   3.6.6
com.apple.driver.DspFuncLib 2.6.0f1
com.apple.vecLib.kext   1.0.0
com.apple.iokit.IOAudioFamily   1.9.5fc2
com.apple.kext.OSvKernDSPLib    1.14
com.apple.iokit.IOBluetoothHostControllerUSBTransport   4.2.3f10
com.apple.iokit.IOFireWireIP    2.2.6
com.apple.driver.IOPlatformPluginLegacy 1.0.0
com.apple.driver.AppleSMBusController   1.0.11d1
com.apple.driver.IOPlatformPluginFamily 5.7.0d10
com.apple.driver.AppleHDAController 2.6.0f1
com.apple.iokit.IOHDAFamily 2.6.0f1
com.apple.driver.AppleSMBusPCI  1.0.12d1
com.apple.iokit.IOSurface   91
com.apple.iokit.IOBluetoothFamily   4.2.3f10
com.apple.driver.AppleSMC   3.1.8
com.apple.iokit.IOAcceleratorFamily 98.14
com.apple.kext.AMDSupport   1.2.0
com.apple.AppleGraphicsDeviceControl    3.4.35
com.apple.iokit.IONDRVSupport   2.4.1
com.apple.iokit.IOGraphicsFamily    2.4.1
com.apple.driver.AppleUSBMergeNub   650.4.0
com.apple.iokit.IOUSBHIDDriver  660.4.0
com.apple.driver.AppleUSBComposite  656.4.1
com.apple.driver.AppleThunderboltDPInAdapter    3.1.7
com.apple.driver.AppleThunderboltDPOutAdapter   3.1.7
com.apple.driver.AppleThunderboltDPAdapterFamily    3.1.7
com.apple.driver.AppleThunderboltPCIUpAdapter   1.4.5
com.apple.driver.AppleThunderboltPCIDownAdapter 1.4.5
com.apple.driver.CoreStorage    380
com.apple.driver.AppleThunderboltNHI    2.0.1
com.apple.iokit.IOThunderboltFamily 3.2.7
com.apple.iokit.IOFireWireFamily    4.5.5
com.apple.iokit.IOUSBUserClient 660.4.2
com.apple.iokit.IOAHCIFamily    2.6.5
com.apple.iokit.IO80211Family   630.35
com.apple.iokit.IOEthernetAVBController 1.0.3b4
com.apple.driver.mDNSOffloadUserClient  1.0.1b5
com.apple.iokit.IONetworkingFamily  3.2
com.apple.iokit.IOUSBFamily 675.4.0
com.apple.driver.AppleEFINVRAM  2.0
com.apple.driver.AppleEFIRuntime    2.0
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.iokit.IOSMBusFamily   1.1
com.apple.security.sandbox  278.11
com.apple.kext.AppleMatch   1.0.0d1
com.apple.security.TMSafetyNet  7
com.apple.driver.AppleKeyStore  2
com.apple.driver.DiskImages 371.1
com.apple.iokit.IOStorageFamily 1.9
com.apple.iokit.IOReportFamily  23
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleACPIPlatform  2.0
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOACPIFamily    1.4
com.apple.kec.corecrypto    1.0
com.apple.kec.pthread   1
Model: Macmini5,2, BootROM MM51.0077.B10, 2 processors, Intel Core i5, 2.5 GHz, 16 GB, SMC 1.75f0
Graphics: AMD Radeon HD 6630M, AMD Radeon HD 6630M, PCIe, 256 MB
Memory Module: BANK 0/DIMM0, 8 GB, DDR3, 1333 MHz, 0x0000, 0x4B425F38475F44335F313333335F43390000
Memory Module: BANK 1/DIMM0, 8 GB, DDR3, 1333 MHz, 0x0000, 0x4B425F38475F44335F313333335F43390000
AirPort: spairport_wireless_card_type_airport_extreme (0x14E4, 0xE4), Broadcom BCM43xx 1.0 (5.106.98.100.22)
Bluetooth: Version 4.2.3f10 13477, 3 services, 15 devices, 1 incoming serial ports
Network Service: Ethernet, Ethernet, en0
PCI Card: pci10b5,8609, Thunderbolt@192,0,1
PCI Card: pci1b21,612, AHCI Controller, Thunderbolt@194,0,0
PCI Card: pci1b21,612, AHCI Controller, Thunderbolt@196,0,0
PCI Card: pci1b21,612, AHCI Controller, Thunderbolt@195,0,0
Serial ATA Device: Samsung SSD 840 PRO Series, 128,04 GB
Serial ATA Device: TOSHIBA MK5065GSXF, 500,11 GB
Serial ATA Device: ST2000DM001-1CH164, 2 TB
Serial ATA Device: ST2000DM001-1CH164, 2 TB
Serial ATA Device: ST2000DM001-1CH164, 2 TB
Serial ATA Device: ST2000DM001-1CH164, 2 TB
Serial ATA Device: ST2000DM001-1CH164, 2 TB
USB Device: Hub
USB Device: BRCM20702 Hub
USB Device: Bluetooth USB Host Controller
USB Device: Hub
USB Device: Back-UPS ES 700G FW:871.O1 .I USB FW:O1
USB Device: IR Receiver
Thunderbolt Bus: Mac mini, Apple Inc., 25.1
Thunderbolt Device: 5big SR, LaCie, 3, 26.2

I was copying about 1TB of data from a single disk (JHFS+) on a zdev (JHFS+ formatted) in a raidz1 pool. The platform is a Mac mini with Mavericks 10.9.2 updated. The ZFS package installed is 1.2.0 from o3x.org. The panic report details all my hardware, if you need other information I'll provide it happily.

Michele

[lundman]

It is a known issue that the kernel memory allocator will just panic, rather than give us a chance to clean up. This is one of the reasons the ARC max size is so small on OSX. I would suggest you tune it even smaller to avoid these panics.

You should see a message like this when starting ZFS:

ZFS: ARC limit set to (arc_c_max): 268435456

So to set the same limit use

sysctl -w zfs.arc_max=268435456

We also have git branch "bmalloc" which uses an alternate allocator which do not have the limits imposed on us, it will become the "server" version.

[codutti]

Thanks for the quick answer. I've updated my startup script to execute the sysctl command with my arc_c_max value. Let me test it for a few days and I'll tell if the problem is resolved. One more question: When the server package will be released I'll be able to upgrade from desktop or I'll be forced to uninstall the desktop package?

[rottegift]

The bmalloc branch (as of 31 march) works great with the vnode_threadX branch (before 29 march), even with , for example, sysctl -w zfs.arc_max=4000000000, which kills the master branch dead.

Why not make the bmalloc slice allocator the default ?

[ilovezfs]

That of course is the plan once it has received more testing.

As for vnode_threadX, it's already merged back to master.

[ilovezfs]

@rottegift FYI, we've decided not to use the vnode_threadX changes and instead put in guard rails to prevent the reclaim list from growing too long. You should move to master.

[rottegift]

On 2 Apr, 2014, at 10:54, ilovezfs notifications@github.com wrote:

@rottegift FYI, we've decided not to use the vnode_threadX changes and instead put in guard rails to prevent the reclaim list from growing too long. You should move to master.

Done. Master+bmalloc is up and running fine.

[ilovezfs]

@rottegift Great.