rails / solid_queue

Database-backed Active Job backend
MIT License
1.95k stars 130 forks source link

MacOS crash on solid_queue:start #416

Closed yoniamir closed 4 days ago

yoniamir commented 4 days ago

I'm running MacOS 15.1 on the new MacOS 16'' Pro (M4 Pro chip).

I've upgrade my repo from Rails 7.2 to Rails 8.0.0 and as soon as I migrated to SolidQueue, my computer completely crashes every single time I run the command: solid_queue:start.

At first I thought it was some fluke, but this is happening consistency every single time and I cannot find a way to get around it. Here's the crash log from MacOS that shows at the end of the reboot:

panic(cpu 10 caller 0xfffffe004f38400c): [SPTM] VIOLATION_ILLEGAL_SPRR_INDEX: sptm_map_page(sptm.c:406) - page_fte(0xfffffdf00008fae0), page_fte->type(0xe), new_sprr_index(0x7), new_pte(0x60010025fc0e43), vaddr(0x1184f8000)

Debugger message: panic
Memory ID: 0xff
OS release type: User
OS version: 24B2083
Kernel version: Darwin Kernel Version 24.1.0: Thu Oct 10 21:06:57 PDT 2024; root:xnu-11215.41.3~3/RELEASE_ARM64_T6041
Fileset Kernelcache UUID: 918FFAAEF5863665571822DC0F4C5BC0
Kernel UUID: 58CD7EDF-BF0B-3375-B9F8-F2484179E8A1
Boot session UUID: FC8666DF-1476-41BD-A000-54C7A42A0317
iBoot version: iBoot-11881.41.5
secure boot?: YES
roots installed: 0
Paniclog version: 14
Debug Header address: 0xfffffe002a8a1000
Debug Header entry count: 3
TXM load address: 0xfffffe003a820000
TXM UUID: 772CA8A2-ABBC-34D1-8CC3-C295F9A174FE
Debug Header kernelcache load address: 0xfffffe004a820000
Debug Header kernelcache UUID: 918FFAAE-F586-3665-5718-22DC0F4C5BC0
SPTM load address: 0xfffffe002a820000
SPTM UUID: 9EF80C6B-0E45-300A-8EC1-8C49E7CA3D3B
KernelCache slide: 0x000000004381c000
KernelCache base:  0xfffffe004a820000
Kernel slide:      0x0000000043824000
Kernel text base:  0xfffffe004a828000
Kernel text exec slide: 0x0000000044ee0000
Kernel text exec base:  0xfffffe004bee4000
mach_absolute_time: 0x971482b7
Epoch Time:        sec       usec
  Boot    : 0x673a19b9 0x00097db4
  Sleep   : 0x00000000 0x00000000
  Wake    : 0x00000000 0x00000000
  Calendar: 0x673a1a1c 0x000d677f

Zone info:
  Zone map: 0xfffffe1000050000 - 0xfffffe3000050000
  . VM    : 0xfffffe1000050000 - 0xfffffe14ccd1c000
  . RO    : 0xfffffe14ce000000 - 0xfffffe1668000000
  . GEN0  : 0xfffffe1668000000 - 0xfffffe1b3480c000
  . GEN1  : 0xfffffe1b3480c000 - 0xfffffe2001018000
  . GEN2  : 0xfffffe2001018000 - 0xfffffe24cd828000
  . GEN3  : 0xfffffe24cd828000 - 0xfffffe299a038000
  . DATA  : 0xfffffe299a038000 - 0xfffffe3000050000
  Metadata: 0xfffffe4232864000 - 0xfffffe423a864000
  Bitmaps : 0xfffffe423a864000 - 0xfffffe423ede0000
  Extra   : 0 - 0

CORE 0 recently retired instr at 0x0000000000000000
CORE 1 recently retired instr at 0x0000000000000000
CORE 2 recently retired instr at 0x0000000000000000
CORE 3 recently retired instr at 0x0000000000000000
CORE 4 is offline, skipping
CORE 5 is offline, skipping
CORE 6 is offline, skipping
CORE 7 is offline, skipping
CORE 8 is offline, skipping
CORE 9 recently retired instr at 0x0000000000000000
CORE 10 recently retired instr at 0x0000000000000000
CORE 11 recently retired instr at 0x0000000000000000
CORE 12 recently retired instr at 0x0000000000000000
CORE 13 recently retired instr at 0x0000000000000000
TPIDRx_ELy = {1: 0xfffffe1ffe688f70  0: 0x000000000000200a  0ro: 0x00000001fb427ba0 }
CORE 0: PC=0xfffffe004ee2abf0, LR=0xfffffe004ee2abf0, FP=0xfffffe425ec6b840
CORE 1: PC=0x00000002401e21d8, LR=0x0000000000000001, FP=0x000000016b3696a0
CORE 2: PC=0x000000010b31b2a0, LR=0x000000010b31b328, FP=0x0000000170b21a20
CORE 3: PC=0xfffffe004bfb2dbc, LR=0xfffffe004bf0da70, FP=0xfffffe425ee4bbc0
CORE 4: PC=0xfffffe004c08d138, LR=0xfffffe004c097e24, FP=0xfffffe425ec83e40
CORE 5: PC=0x00000001030cdc04, LR=0x00000001030cdbac, FP=0x000000016cfdf150
CORE 6: PC=0x00000001a7349894, LR=0x00000001a7336324, FP=0x000000016fd64880
CORE 7: PC=0x00000001963537dc, LR=0x00000001963537dc, FP=0x000000016f057680
CORE 8: PC=0x00000001120d9110, LR=0x00000001121c7a58, FP=0x000000016d490970
CORE 9: PC=0xfffffe004c08d15c, LR=0xfffffe004c097e24, FP=0xfffffe425ec9be40
CORE 10 is the one that panicked. Check the full backtrace for details.
CORE 11: PC=0xfffffe004f384cb4, LR=0xfffffe004c077cb4, FP=0xfffffe425ed97860
CORE 12: PC=0xfffffe004c08d138, LR=0xfffffe004c097e24, FP=0xfffffe425ecbfe40
CORE 13: PC=0xfffffe004bf714cc, LR=0xfffffe004bf714cc, FP=0xfffffe425d147ed0
Compressor Info: 0% of compressed pages limit (OK) and 0% of segments limit (OK) with 0 swapfiles and OK swap space
Panicked task 0xfffffe200178cf18: 1685 pages, 2 threads: pid 3238: ruby
Panicked thread: 0xfffffe1ffe688f70, backtrace: 0xfffffe425ef831e0, tid: 17983
          lr: 0xfffffe004bf34c60  fp: 0xfffffe425ef83270
          lr: 0xfffffe004c093f34  fp: 0xfffffe425ef832e0
          lr: 0xfffffe004c092330  fp: 0xfffffe425ef833a0
          lr: 0xfffffe004bee9d18  fp: 0xfffffe425ef833b0
          lr: 0xfffffe004bf34564  fp: 0xfffffe425ef837a0
          lr: 0xfffffe004c7a3370  fp: 0xfffffe425ef837c0
          lr: 0xfffffe004f38400c  fp: 0xfffffe425ef837d0
          lr: 0xfffffe002a8e1cd8  fp: 0xfffffdf00244bed0
      Kernel Extensions in backtrace:
         com.apple.sptm(24.1)[9EF80C6B-0E45-300A-8EC1-8C49E7CA3D3B]@0xfffffe002a898000->0xfffffe002a8e3fff

last started kext at 718822501: com.apple.filesystems.autofs    3.0 (addr 0xfffffe004b3410b0, size 5847)
loaded kexts:
com.apple.filesystems.autofs    3.0
com.apple.UVCService    1
com.apple.driver.AppleHIDALSService 1
com.apple.driver.AppleTopCaseHIDEventDriver 8410.3
com.apple.driver.AppleBiometricServices 1
com.apple.driver.CoreKDL    1
com.apple.driver.DiskImages.ReadWriteDiskImage  493.0.0
com.apple.driver.DiskImages.UDIFDiskImage   493.0.0
com.apple.driver.DiskImages.RAMBackingStore 493.0.0
com.apple.driver.DiskImages.FileBackingStore    493.0.0
com.apple.driver.AppleALSColorSensor    1.0.0d1
com.apple.driver.AppleSmartBatteryManager   161.0.0
com.apple.driver.AppleThunderboltIP 4.0.3
com.apple.driver.AppleUSBDeviceNCM  5.0.0
com.apple.driver.SEPHibernation 1
com.apple.driver.AppleSmartIO2  1
com.apple.AppleEmbeddedSimpleSPINORFlasher  1
com.apple.driver.ApplePMP   1
com.apple.driver.AppleTypeCRetimer  1.0.0
com.apple.driver.AppleCS42L84Audio  810.2
com.apple.driver.AppleSN012776Amp   810.2
com.apple.driver.AppleFileSystemDriver  3.0.1
com.apple.nke.l2tp  1.9
com.apple.filesystems.tmpfs 1
com.apple.filesystems.nfs   1
com.apple.filesystems.lifs  1
com.apple.filesystems.apfs  2313.41.1
com.apple.IOTextEncryptionFamily    1.0.0
com.apple.filesystems.hfs.kext  674
com.apple.security.BootPolicy   1
com.apple.BootCache 40
com.apple.AppleFSCompression.AppleFSCompressionTypeZlib 1.0.0
com.apple.AppleFSCompression.AppleFSCompressionTypeDataless 1.0.0d1
com.apple.driver.AppleBluetoothModule   1
com.apple.driver.AppleBCMWLANBusInterfacePCIe   1
com.apple.driver.AppleEventLogHandler   1
com.apple.driver.AppleT6041CLPC 1
com.apple.driver.AppleT6041SOCTuner 1
com.apple.driver.AppleT6041PMGR 1
com.apple.driver.ApplePMPFirmware   1
com.apple.driver.AppleMobileDispT604X-DCP   140.0
com.apple.driver.AppleJPEGDriver    7.1.5
com.apple.driver.AppleS8000DWI  1.0.0d1
com.apple.driver.AppleAVE2  803.36.1
com.apple.driver.AppleProResHW  426.1
com.apple.driver.ApplePMUFirmwareDriver 1.0.0
com.apple.driver.AppleDPDisplayTCON 1
com.apple.driver.AppleT6041MCC  1
com.apple.driver.AppleInterruptControllerV3 1.0.0d1
com.apple.AGXG16X   322.10
com.apple.driver.AppleAVD   807
com.apple.driver.AppleS8000AES  1
com.apple.driver.AppleSamsungSerial 1.0.0d1
com.apple.driver.AudioDMAController-T604x   400.97
com.apple.driver.AppleSerialShim    1
com.apple.driver.AppleQSPIMC    1
com.apple.driver.AppleSecureRepair  1.0.0
com.apple.driver.usb.AppleSynopsysUSB40XHCI 1
com.apple.driver.AppleS5L8920XPWM   1.0.0d1
com.apple.driver.AppleT8110DART 1
com.apple.driver.AppleSPIMC 1
com.apple.driver.AppleS5L8940XI2C   1.0.0d2
com.apple.driver.AppleT6041 1
com.apple.driver.AppleM68Buttons    1.0.0d1
com.apple.driver.AppleSDXC  3.5.3
com.apple.iokit.IOUserEthernet  1.0.1
com.apple.driver.usb.AppleUSBUserHCI    1
com.apple.iokit.IOKitRegistryCompatibility  1
com.apple.iokit.EndpointSecurity    1
com.apple.driver.AppleDiskImages2   379.0.11
com.apple.AppleSystemPolicy 2.0.0
com.apple.kec.InvalidateHmac    1
com.apple.kec.AppleEncryptedArchive 1
com.apple.driver.driverkit.serial   6.0.0
com.apple.driver.usb.IOUSBHostHIDDevice 1.2
com.apple.driver.usb.cdc.ecm    5.0.0
com.apple.driver.usb.cdc.acm    5.0.0
com.apple.driver.usb.serial 6.0.0
com.apple.driver.usb.cdc.ncm    5.0.0
com.apple.driver.usb.cdc    5.0.0
com.apple.driver.AppleUSBAudio  701.72
com.apple.iokit.IOAudioFamily   600.2
com.apple.vecLib.kext   1.2.0
com.apple.driver.usb.AppleUSBXHCIPCI    1.2
com.apple.driver.AppleThunderboltDPOutAdapter   8.5.1
com.apple.driver.AppleThunderboltPCIUpAdapter   4.1.1
com.apple.driver.AppleHSBluetoothDriver 8410.3
com.apple.driver.IOBluetoothHIDDriver   9.0.0
com.apple.driver.AppleActuatorDriver    8410.3
com.apple.driver.AppleMultitouchDriver  8410.3
com.apple.driver.AppleHIDKeyboard   8010.1
com.apple.driver.AppleMesaSEPDriver 100.99
com.apple.iokit.IOBiometricFamily   1
com.apple.driver.DiskImages.KernelBacked    493.0.0
com.apple.iokit.IOAVBFamily 1300.26
com.apple.driver.AppleXsanScheme    3
com.apple.driver.IOPAudioVoiceTriggerDevice 400.40
com.apple.driver.AppleConvergedIPCOLYBTControl  1
com.apple.driver.AppleConvergedPCI  1
com.apple.driver.AppleBluetoothDebug    1
com.apple.driver.IOPAudioIOBufferDevice 200.39
com.apple.driver.AppleUVDMDriver    1.0.0
com.apple.driver.AppleUVDM  1.0.0
com.apple.driver.AppleIOPADMAStream 200.17
com.apple.driver.IOPAudioLPMicDevice    200.39
com.apple.driver.IOPAudioClientManagerDevice    200.39
com.apple.driver.AppleBTM   1.0.1
com.apple.driver.AOPAudio2  200.11
com.apple.driver.usb.networking 5.0.0
com.apple.driver.AppleThunderboltDPInAdapter    8.5.1
com.apple.driver.AppleThunderboltDPAdapterFamily    8.5.1
com.apple.driver.AppleThunderboltUSBDownAdapter 1.0.4
com.apple.driver.AppleThunderboltPCIDownAdapter 4.1.1
com.apple.driver.AppleDCPDPTXProxy  1.0.0
com.apple.driver.DCPDPFamilyProxy   1
com.apple.driver.AppleSEPHDCPManager    1.0.1
com.apple.driver.AppleTrustedAccessory  1
com.apple.iokit.AppleSEPGenericTransfer 1
com.apple.driver.AppleDiagnosticDataAccessReadOnly  1.0.0
com.apple.plugin.IOgPTPPlugin   1300.51
com.apple.driver.AppleCSEmbeddedAudio   810.2
com.apple.driver.AppleEmbeddedAudio 810.2
com.apple.iokit.AppleARMIISAudio    400.22
com.apple.driver.IISAudioIsolatedStreamECProxy  400.22
com.apple.nke.ppp   1.9
com.apple.driver.AppleBSDKextStarter    3
com.apple.kext.triggers 1.0
com.apple.driver.IOHIDPowerSource   1
com.apple.driver.AppleCallbackPowerSource   1
com.apple.filesystems.hfs.encodings.kext    1
com.apple.driver.AppleSyntheticGameController   12.1.3
com.apple.driver.AppleHPM   3.4.4
com.apple.driver.AppleStockholmControl  1.0.0
com.apple.driver.AppleBluetoothDebugService 1
com.apple.driver.AppleBCMWLANCore   1.0.0
com.apple.iokit.IO80211Family   1200.13.1
com.apple.driver.IOImageLoader  1.0.0
com.apple.driver.AppleOLYHAL    1
com.apple.driver.ApplePassthroughPPM    3.0
com.apple.driver.ApplePMGR  1
com.apple.iokit.IONVMeFamily    2.1.0
com.apple.driver.AppleNANDConfigAccess  1.0.0
com.apple.iokit.IOMobileGraphicsFamily-DCP  343.0.0
com.apple.iokit.IOMobileGraphicsFamily  343.0.0
com.apple.driver.AppleM2ScalerCSCDriver 265.0.0
com.apple.driver.DCPAVFamilyProxy   1
com.apple.driver.AppleDisplayCrossbar   1.0.0
com.apple.iokit.IODisplayPortFamily 1.0.0
com.apple.driver.AppleH16ANEInterface   8.107.2
com.apple.driver.AppleDCP   1
com.apple.driver.AppleARMWatchdogTimer  1
com.apple.driver.AppleSART  1
com.apple.driver.ApplePTD   1.0.0
com.apple.AGXFirmwareKextG16XRTBuddy    1
com.apple.AGXFirmwareKextRTBuddy64  322.10
com.apple.driver.AppleSPMIPMU   1.0.1
com.apple.driver.AppleDialogPMU 1.0.1
com.apple.driver.AppleHIDTransportFIFO  8000.26
com.apple.driver.AppleHIDTransport  8000.26
com.apple.driver.AppleSPU   1
com.apple.driver.AppleInputDeviceSupport    8000.26
com.apple.driver.AppleFirmwareKit   1
com.apple.iokit.IOGPUFamily 104.1.2
com.apple.driver.DMAChannelProxy    400.97
com.apple.driver.ExclavesAudioKext  200.54
com.apple.driver.AudioSharedDARTMapperProxy 200.54
com.apple.driver.AudioDMAFamily 400.97
com.apple.driver.AudioDMACLLTEscalationDetector-Stub    400.97
com.apple.driver.AppleThunderboltNHI    7.2.81
com.apple.driver.AppleUSBXDCIARM    1.0
com.apple.driver.AppleUSBXDCI   1.0
com.apple.iokit.IOUSBDeviceFamily   2.0.0
com.apple.driver.usb.AppleSynopsysUSBXHCI   1
com.apple.driver.usb.AppleUSBXHCI   1.2
com.apple.driver.AppleEmbeddedUSBHost   1
com.apple.driver.usb.AppleUSBHub    1.2
com.apple.driver.usb.AppleUSBHostCompositeDevice    1.2
com.apple.driver.AppleT6040TypeCPhy 1
com.apple.driver.AppleTypeCPhy  1
com.apple.driver.AppleSPMI  1.0.1
com.apple.driver.AppleA7IOP-ASCWrap-v6  1.0.2
com.apple.driver.AppleT6040PCIeC    1
com.apple.iokit.IOThunderboltFamily 9.3.3
com.apple.iokit.IOPortFamily    1.0
com.apple.driver.ApplePIODMA    1
com.apple.driver.usb.AppleUSBHostPacketFilter   1.0
com.apple.driver.AppleMobileApNonce 1
com.apple.driver.AppleDockChannel   1
com.apple.driver.AppleA7IOP-MXWrap-v1   1.0.2
com.apple.driver.AppleT6040PCIe 1
com.apple.driver.AppleH16CameraInterface    3.111.0
com.apple.iokit.IOPAudioDriverFamily    200.11
com.apple.driver.AppleEmbeddedAudioLibs 400.9
com.apple.driver.AppleFirmwareUpdateKext    1
com.apple.driver.AppleGPIOICController  1.0.2
com.apple.driver.AppleEverestErrorHandler   1
com.apple.driver.AppleMultiFunctionManager  1
com.apple.driver.corecapture    1.0.4
com.apple.driver.AppleEmbeddedPCIE  1
com.apple.iokit.IOTimeSyncFamily    1300.51
com.apple.driver.DiskImages 493.0.0
com.apple.iokit.IOGraphicsFamily    598
com.apple.iokit.IOBluetoothFamily   9.0.0
com.apple.driver.AppleUIO   1
com.apple.driver.AppleT6041ANEHAL   8.107.3
com.apple.driver.AppleSSE   1.0
com.apple.driver.AppleSEPKeyStore   2
com.apple.driver.AppleUSBTDM    556
com.apple.iokit.IOUSBMassStorageDriver  250
com.apple.iokit.IOPCIFamily 2.9
com.apple.iokit.IOUSBHostFamily 1.2
com.apple.driver.AppleUSBHostMergeProperties    1.2
com.apple.driver.usb.AppleUSBCommon 1.0
com.apple.driver.AppleSMC   3.1.9
com.apple.driver.RTBuddy    1.0.0
com.apple.driver.AppleEmbeddedTempSensor    1.0.0
com.apple.driver.AppleARMPMU    1.0
com.apple.iokit.IOAccessoryManager  1.0.0
com.apple.driver.AppleOnboardSerial 1.0
com.apple.iokit.IOSerialFamily  11
com.apple.iokit.IOSCSIBlockCommandsDevice   498
com.apple.iokit.IOSCSIArchitectureModelFamily   498
com.apple.driver.AppleRSMChannel    1
com.apple.iokit.IORSMFamily 1
com.apple.driver.AppleLockdownMode  1
com.apple.driver.AppleIPAppender    1.0
com.apple.iokit.IOSkywalkFamily 1.0
com.apple.driver.mDNSOffloadUserClient  1.0.1b8
com.apple.iokit.IONetworkingFamily  3.4
com.apple.driver.AppleFDEKeyStore   28.30
com.apple.driver.AppleEffaceableStorage 1.0
com.apple.driver.AppleCredentialManager 1.0
com.apple.driver.AppleSEPManager    1.0.1
com.apple.driver.IODARTFamily   1
com.apple.driver.AppleA7IOP 1.0.2
com.apple.driver.IOSlaveProcessor   1
com.apple.driver.AppleBiometricSensor   2
com.apple.iokit.IOHIDFamily 2.0.0
com.apple.AUC   1.0
com.apple.iokit.IOSurface   372.2.3
com.apple.iokit.IOAVFamily  1.0.0
com.apple.iokit.IOHDCPFamily    1.0.0
com.apple.iokit.IOCECFamily 1
com.apple.iokit.IOAudio2Family  1.0
com.apple.driver.AppleIISController 400.1
com.apple.driver.AppleAudioClockLibs    400.9
com.apple.driver.FairPlayIOKit  72.10.0
com.apple.driver.AppleARMPlatform   1.0.2
com.apple.iokit.IOSlowAdaptiveClockingFamily    1.0.0
com.apple.iokit.IOReportFamily  47
com.apple.security.quarantine   4
com.apple.security.sandbox  300.0
com.apple.iokit.IOStorageFamily 2.1
com.apple.kext.AppleMatch   1.0.0d1
com.apple.driver.AppleMobileFileIntegrity   1.0.5
com.apple.iokit.CoreAnalyticsFamily 1
com.apple.security.AppleImage4  7.0.0
com.apple.kext.CoreTrust    1
com.apple.iokit.IOCryptoAcceleratorFamily   1.0.1
com.apple.kec.pthread   1
com.apple.kec.Libm  1
com.apple.kec.Compression   1.0
com.apple.kec.corecrypto    14.0

** Stackshot Succeeded ** Bytes Traced 529209 (Uncompressed 1347904) **

Please advise.

codealchemy commented 4 days ago

This looks to be a dupe of https://github.com/rails/solid_queue/issues/411, one suggestion there re: disabling YJIT that might be helpful.

rosa commented 4 days ago

Yes! This is the corresponding report for Ruby. It looks like a bug in macOS.