search

cirruslabs / gitlab-tart-executor

GitLab Runner executor to run jobs in Tart VMs
MIT License
60 stars 5 forks source link

MacMini M1 - keeps crashing #66

Closed dinbtechit closed 8 months ago

dinbtechit commented 8 months ago

Tart keeps crashing on mac mini m1.

image

.gitlab-ci.yml

# You can use any remote Tart image.
# Tart Executor will pull it from the registry
# and use it for creating ephemeral VMs.
image: ghcr.io/cirruslabs/macos-sonoma-base:latest

test:
  # In case you tagged runners that have
  # GitLab Tart Executor configured on them
  tags:
    - macos-m1-1

  script:
    - uname -a

Gitlab output

Running with gitlab-runner 16.9.1 (782c6ecb)
  on macos-m1 YunSgmyJz, system ID: s_aa659e[2](https://git.novascotia.ca/digitalplatformservices/test-macos-runner/-/jobs/218279#L2)8ec80
Resolving secrets
00:00
Preparing the "custom" executor
01:01
Using Custom executor...
2024/0[3](https://git.novascotia.ca/digitalplatformservices/test-macos-runner/-/jobs/218279#L3)/12 18:03:10 Pulling the latest version of ghcr.io/cirruslabs/macos-sonoma-base:latest...
202[4](https://git.novascotia.ca/digitalplatformservices/test-macos-runner/-/jobs/218279#L4)/03/12 1[8](https://git.novascotia.ca/digitalplatformservices/test-macos-runner/-/jobs/218279#L8):03:11 Cloning and configuring a new VM...
2024/03/12 18:03:11 Waiting for the VM to boot and be SSH-able...
2024/03/12 18:04:11 tart command returned non-zero exit code: "no IP address found, is your VM running?"
ERROR: Job failed: exit status 1

Full error:

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               tart [3859]
Path:                  /opt/homebrew/*/tart.app/Contents/MacOS/tart
Identifier:            tart
Version:               ???
Code Type:             ARM-64 (Native)
Parent Process:        gitlab-tart-executor [3856]
Responsible:           gitlab-runner [3854]
User ID:               501

Date/Time:             2024-03-12 18:15:52.8319 -0300
OS Version:            macOS 14.4 (23E214)
Report Version:        12
Anonymous UUID:        9849F718-33B6-6341-7C01-2020ABE3E362

Sleep/Wake UUID:       3293EFDD-DAFF-4E6E-8057-A4FCF5D11F45

Time Awake Since Boot: 7700 seconds

System Integrity Protection: enabled

Crashed Thread:        0

Exception Type:        EXC_BREAKPOINT (SIGTRAP)
Exception Codes:       0x0000000000000001, 0x0000000100ec5904

Termination Reason:    Namespace SIGNAL, Code 5 Trace/BPT trap: 5
Terminating Process:   exc handler [3859]

Thread 0 Crashed:
0   tart                                   0x100ec5904 Run.run() + 6660
1   tart                                   0x100ecc30d protocol witness for AsyncParsableCommand.run() in conformance Run + 1
2   tart                                   0x100f1d175 static Root.main() + 1
3   tart                                   0x100f1dff5 specialized thunk for @escaping @convention(thin) @async () -> () + 1
4   libswift_Concurrency.dylib             0x2524fb149 completeTaskAndRelease(swift::AsyncContext*, swift::SwiftError*) + 1

Thread 1:
0   libsystem_pthread.dylib                0x18be01d20 start_wqthread + 0

Thread 2:
0   libsystem_pthread.dylib                0x18be01d20 start_wqthread + 0

Thread 3:
0   libsystem_pthread.dylib                0x18be01d20 start_wqthread + 0

Thread 0 crashed with ARM Thread State (64-bit):
    x0: 0x000000016f2cdba0   x1: 0x000000016f2cda70   x2: 0x00000001f5a69fc8   x3: 0x0000000000000000
    x4: 0x0000000100fab9b0   x5: 0x0000000000000000   x6: 0x0000000000000000   x7: 0x0000000000000023
    x8: 0x00000001f3b66bc0   x9: 0x0000000000000000  x10: 0x00000000fffffffe  x11: 0x0000000000000002
   x12: 0x0000000000000001  x13: 0x0000000000000001  x14: 0x00000001f5a69fc8  x15: 0x00000001f5a69fc8
   x16: 0x000000019be672fc  x17: 0x000000019be672f0  x18: 0x0000000000000000  x19: 0x0000000128009a30
   x20: 0x0000000000000000  x21: 0x00006000028b5110  x22: 0x0000000128009c90  x23: 0x0000000128009cb0
   x24: 0x0000000128009b40  x25: 0x00000001f3bbd7f8  x26: 0x0000000128009d30  x27: 0x0000000100f34998
   x28: 0x00000001f4187558   fp: 0x000000016f2cdc30   lr: 0x0000000100ec5680
    sp: 0x000000016f2cdad0   pc: 0x0000000100ec5904 cpsr: 0x60001000
   far: 0x0000000000000000  esr: 0xf2000001 (Breakpoint) brk 1

Binary Images:
       0x100b2c000 -        0x101003fff tart (*) <67cac1cf-5130-3c59-b0a0-7a661e5c6dd3> /opt/homebrew/*/tart.app/Contents/MacOS/tart
       0x2524a5000 -        0x252510ff7 libswift_Concurrency.dylib (*) <d6c9a750-a3e9-3aa3-9d68-e0d6b1469648> /usr/lib/swift/libswift_Concurrency.dylib
               0x0 - 0xffffffffffffffff ??? (*) <00000000-0000-0000-0000-000000000000> ???
       0x19babf000 -        0x19c01cfff libswiftCore.dylib (*) <05650d36-bbc5-3760-aec7-4d2751b5d729> /usr/lib/swift/libswiftCore.dylib
       0x18be00000 -        0x18be0cfff libsystem_pthread.dylib (*) <45239f06-cc53-36d0-9933-7776ac7ea2fa> /usr/lib/system/libsystem_pthread.dylib

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=1.0G resident=0K(0%) swapped_out_or_unallocated=1.0G(100%)
Writable regions: Total=944.8M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=944.8M(100%)

                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
Activity Tracing                   256K        1 
ColorSync                          432K       22 
CoreGraphics                        16K        1 
CoreServices                       224K        1 
Kernel Alloc Once                   32K        1 
MALLOC                           934.3M       27 
MALLOC guard page                  192K       12 
STACK GUARD                       56.1M        4 
Stack                             9808K        4 
VM_ALLOCATE                         32K        2 
__AUTH                            1396K      259 
__AUTH_CONST                      23.2M      439 
__CTF                               824        1 
__DATA                             9.9M      426 
__DATA_CONST                      21.4M      444 
__DATA_DIRTY                      1283K      135 
__FONT_DATA                          4K        1 
__LINKEDIT                       531.4M        2 
__OBJC_RO                         71.7M        1 
__OBJC_RW                         2195K        1 
__TEXT                           533.2M      460 
dyld private memory                272K        2 
mapped file                       39.6M        6 
shared memory                      848K       13 
===========                     =======  ======= 
TOTAL                              2.2G     2265 

-----------
Full Report
-----------

{"app_name":"tart","timestamp":"2024-03-12 18:15:52.00 -0300","app_version":"","slice_uuid":"67cac1cf-5130-3c59-b0a0-7a661e5c6dd3","build_version":"","platform":1,"share_with_app_devs":1,"is_first_party":1,"bug_type":"309","os_version":"macOS 14.4 (23E214)","roots_installed":0,"incident_id":"A2F98E7F-0285-434F-883E-54777B9620BD","name":"tart"}
{
  "uptime" : 7700,
  "procRole" : "Unspecified",
  "version" : 2,
  "userID" : 501,
  "deployVersion" : 210,
  "modelCode" : "Macmini9,1",
  "coalitionID" : 1913,
  "osVersion" : {
    "train" : "macOS 14.4",
    "build" : "23E214",
    "releaseType" : "User"
  },
  "captureTime" : "2024-03-12 18:15:52.8319 -0300",
  "codeSigningMonitor" : 1,
  "incident" : "A2F98E7F-0285-434F-883E-54777B9620BD",
  "pid" : 3859,
  "translated" : false,
  "cpuType" : "ARM-64",
  "roots_installed" : 0,
  "bug_type" : "309",
  "procLaunch" : "2024-03-12 18:15:52.7799 -0300",
  "procStartAbsTime" : 185394158884,
  "procExitAbsTime" : 185395399017,
  "procName" : "tart",
  "procPath" : "\/opt\/homebrew\/*\/tart.app\/Contents\/MacOS\/tart",
  "parentProc" : "gitlab-tart-executor",
  "parentPid" : 3856,
  "coalitionName" : "gitlab-runner",
  "crashReporterKey" : "9849F718-33B6-6341-7C01-2020ABE3E362",
  "responsiblePid" : 3854,
  "responsibleProc" : "gitlab-runner",
  "codeSigningID" : "tart",
  "codeSigningTeamID" : "9M2P8L4D89",
  "codeSigningFlags" : 570522385,
  "codeSigningValidationCategory" : 6,
  "codeSigningTrustLevel" : 4294967295,
  "instructionByteStream" : {"beforePC":"GPEBlOBTQPkW8QGU4AMaqhTxAZTABkD5FACA0pT6\/xcgACDUIAAg1A==","atPC":"IAAg1PwPGvj7awGp+WMCqfdbA6n0TwSp\/XsFqf1DAZH\/wwTRqEMB0Q=="},
  "sleepWakeUUID" : "3293EFDD-DAFF-4E6E-8057-A4FCF5D11F45",
  "sip" : "enabled",
  "exception" : {"codes":"0x0000000000000001, 0x0000000100ec5904","rawCodes":[1,4310456580],"type":"EXC_BREAKPOINT","signal":"SIGTRAP"},
  "termination" : {"flags":0,"code":5,"namespace":"SIGNAL","indicator":"Trace\/BPT trap: 5","byProc":"exc handler","byPid":3859},
  "os_fault" : {"process":"tart"},
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 0,
  "threads" : [{"triggered":true,"id":62157,"threadState":{"x":[{"value":6160178080},{"value":6160177776},{"value":8416305096,"symbolLocation":0,"symbol":"type metadata for Bool"},{"value":0},{"value":4311398832,"symbolLocation":0,"symbol":"nominal type descriptor for Parsed"},{"value":0},{"value":0},{"value":35},{"value":8383785920,"symbolLocation":0,"symbol":"NSApp"},{"value":0},{"value":4294967294},{"value":2},{"value":1},{"value":1},{"value":8416305096,"symbolLocation":0,"symbol":"type metadata for Bool"},{"value":8416305096,"symbolLocation":0,"symbol":"type metadata for Bool"},{"value":6910538492,"symbolLocation":368,"symbol":"swift_getEnumCaseMultiPayload"},{"value":6910538480,"symbolLocation":356,"symbol":"swift_getEnumCaseMultiPayload"},{"value":0},{"value":4966095408},{"value":0},{"value":105553158951184},{"value":4966096016},{"value":4966096048},{"value":4966095680},{"value":8384141304,"symbolLocation":6088,"symbol":"InitialAllocationPool"},{"value":4966096176},{"value":4310911384,"symbolLocation":0,"symbol":"type metadata accessor for VMDirectory"},{"value":8390210904,"symbolLocation":0,"symbol":"_swiftEmptyArrayStorage"}],"flavor":"ARM_THREAD_STATE64","lr":{"value":4310455936},"cpsr":{"value":1610616832},"fp":{"value":6160178224},"sp":{"value":6160177872},"esr":{"value":4060086273,"description":"(Breakpoint) brk 1"},"pc":{"value":4310456580,"matchesCrashFrame":1},"far":{"value":0}},"frames":[{"imageOffset":3774724,"symbol":"Run.run()","symbolLocation":6660,"imageIndex":0},{"imageOffset":3801869,"symbol":"protocol witness for AsyncParsableCommand.run() in conformance Run","symbolLocation":1,"imageIndex":0},{"imageOffset":4133237,"symbol":"static Root.main()","symbolLocation":1,"imageIndex":0},{"imageOffset":4136949,"symbol":"specialized thunk for @escaping @convention(thin) @async () -> ()","symbolLocation":1,"imageIndex":0},{"imageOffset":352585,"symbol":"completeTaskAndRelease(swift::AsyncContext*, swift::SwiftError*)","symbolLocation":1,"imageIndex":1}]},{"id":62161,"frames":[{"imageOffset":7456,"symbol":"start_wqthread","symbolLocation":0,"imageIndex":4}],"threadState":{"x":[{"value":6160756736},{"value":5891},{"value":6160220160},{"value":0},{"value":409603},{"value":18446744073709551615},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0}],"flavor":"ARM_THREAD_STATE64","lr":{"value":0},"cpsr":{"value":4096},"fp":{"value":0},"sp":{"value":6160756736},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6641687840},"far":{"value":0}}},{"id":62164,"frames":[{"imageOffset":7456,"symbol":"start_wqthread","symbolLocation":0,"imageIndex":4}],"threadState":{"x":[{"value":6161330176},{"value":9987},{"value":6160793600},{"value":0},{"value":409604},{"value":18446744073709551615},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0}],"flavor":"ARM_THREAD_STATE64","lr":{"value":0},"cpsr":{"value":4096},"fp":{"value":0},"sp":{"value":6161330176},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6641687840},"far":{"value":0}}},{"id":62176,"frames":[{"imageOffset":7456,"symbol":"start_wqthread","symbolLocation":0,"imageIndex":4}],"threadState":{"x":[{"value":6161903616},{"value":16899},{"value":6161367040},{"value":0},{"value":409603},{"value":18446744073709551615},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0},{"value":0}],"flavor":"ARM_THREAD_STATE64","lr":{"value":0},"cpsr":{"value":4096},"fp":{"value":0},"sp":{"value":6161903616},"esr":{"value":1442840704,"description":" Address size fault"},"pc":{"value":6641687840},"far":{"value":0}}}],
  "usedImages" : [
  {
    "source" : "P",
    "arch" : "arm64",
    "base" : 4306681856,
    "size" : 5079040,
    "uuid" : "67cac1cf-5130-3c59-b0a0-7a661e5c6dd3",
    "path" : "\/opt\/homebrew\/*\/tart.app\/Contents\/MacOS\/tart",
    "name" : "tart"
  },
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" : 9970536448,
    "size" : 442360,
    "uuid" : "d6c9a750-a3e9-3aa3-9d68-e0d6b1469648",
    "path" : "\/usr\/lib\/swift\/libswift_Concurrency.dylib",
    "name" : "libswift_Concurrency.dylib"
  },
  {
    "size" : 0,
    "source" : "A",
    "base" : 0,
    "uuid" : "00000000-0000-0000-0000-000000000000"
  },
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" : 6906703872,
    "size" : 5627904,
    "uuid" : "05650d36-bbc5-3760-aec7-4d2751b5d729",
    "path" : "\/usr\/lib\/swift\/libswiftCore.dylib",
    "name" : "libswiftCore.dylib"
  },
  {
    "source" : "P",
    "arch" : "arm64e",
    "base" : 6641680384,
    "size" : 53248,
    "uuid" : "45239f06-cc53-36d0-9933-7776ac7ea2fa",
    "path" : "\/usr\/lib\/system\/libsystem_pthread.dylib",
    "name" : "libsystem_pthread.dylib"
  }
],
  "sharedCache" : {
  "base" : 6637223936,
  "size" : 4189880320,
  "uuid" : "664c17e3-f7ed-3226-8594-589aa9bf8fd2"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=1.0G resident=0K(0%) swapped_out_or_unallocated=1.0G(100%)\nWritable regions: Total=944.8M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=944.8M(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nActivity Tracing                   256K        1 \nColorSync                          432K       22 \nCoreGraphics                        16K        1 \nCoreServices                       224K        1 \nKernel Alloc Once                   32K        1 \nMALLOC                           934.3M       27 \nMALLOC guard page                  192K       12 \nSTACK GUARD                       56.1M        4 \nStack                             9808K        4 \nVM_ALLOCATE                         32K        2 \n__AUTH                            1396K      259 \n__AUTH_CONST                      23.2M      439 \n__CTF                               824        1 \n__DATA                             9.9M      426 \n__DATA_CONST                      21.4M      444 \n__DATA_DIRTY                      1283K      135 \n__FONT_DATA                          4K        1 \n__LINKEDIT                       531.4M        2 \n__OBJC_RO                         71.7M        1 \n__OBJC_RW                         2195K        1 \n__TEXT                           533.2M      460 \ndyld private memory                272K        2 \nmapped file                       39.6M        6 \nshared memory                      848K       13 \n===========                     =======  ======= \nTOTAL                              2.2G     2265 \n",
  "legacyInfo" : {
  "threadTriggered" : {

  }
},
  "logWritingSignature" : "3a14963f0175673822938167e52ccd42041e94ee",
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "5ffde50ce2aacd000d47a95f",
      "factorPackIds" : {

      },
      "deploymentId" : 240000364
    },
    {
      "rolloutId" : "63a0a6cfd1cdb67d083006d0",
      "factorPackIds" : {

      },
      "deploymentId" : 240000003
    }
  ],
  "experiments" : [

  ]
}
}

Model: Macmini9,1, BootROM 10151.101.3, proc 8:4:4 processors, 8 GB, SMC 
Graphics: Apple M1, Apple M1, Built-In
Display: LS34A650U, 1920 x 1080 (1080p FHD - Full High Definition), Main, MirrorOff, Online
Memory Module: LPDDR4, Micron
AirPort: spairport_wireless_card_type_wifi (0x14E4, 0x4378), wl0: Jan 13 2024 03:06:59 version 18.20.440.9.7.8.167 FWID 01-7be24857
AirPort: 
Bluetooth: Version (null), 0 services, 0 devices, 0 incoming serial ports
Network Service: Wi-Fi, AirPort, en1
USB Device: USB31Bus
USB Device: USB31Bus
USB Device: USB3.0 Hub
USB Device: USB 10/100/1000 LAN
USB Device: USB2.0 Hub
USB Device: USB-C multiport device
USB Device: USB30Bus
USB Device: Dell KB216 Wired Keyboard
Thunderbolt Bus: Mac mini, Apple Inc.
Thunderbolt Bus: Mac mini, Apple Inc.
fkorotkov commented 8 months ago

Which version of Tart are you using? Please make sure it's not 2.7.0 which had a regression. 2.7.1 includes a fix.

dinbtechit commented 8 months ago

ohh!! it was indeed 2.7.0. Let me try upgrading it to 2.7.1

dinbtechit commented 8 months ago

After upgrading to 2.7.1, tart no longer crashes!

However, strangely I got a new error this time. Any idea, what might be causing this? - tart command returned non-zero exit code: "failed to parse remote name: token recognition error at: ' ' (character 46)"

Gitlab-ci error log

Running with gitlab-runner 16.9.1 (782c6ecb)
  on macos-m1-digitalplatformservices YunSgmyJz, system ID: s_aa659e8ec80
Resolving secrets
00:00
Preparing the "custom" executor
00:00
Using Custom executor...
2024/03/12 19:07:36 Pulling the latest version of ghcr.io/cirruslabs/macos-ventura-xcode:latest ventura-xcode...
2024/03/12 19:07:36 tart command returned non-zero exit code: "failed to parse remote name: token recognition error at: ' ' (character 46)"
WARNING: Cleanup script failed: exit status 1
ERROR: Job failed: exit status 1
fkorotkov commented 8 months ago

It seems you set image to ghcr.io/cirruslabs/macos-ventura-xcode:latest ventura-xcode and it should be just ghcr.io/cirruslabs/macos-ventura-xcode:latest

dinbtechit commented 8 months ago

oops! my bad - that was it. Thank you so much for all the help.. It worked like a charm.

Running with gitlab-runner 16.9.1 (782c6ecb)
  on macos-m1 YunSgmyJz, system ID: s_aa659e[2](https://git.novascotia.ca/digitalplatformservices/test-macos-runner/-/jobs/218302#L2)8ec80
Resolving secrets
00:00
Preparing the "custom" executor
00:12
Using Custom executor...
2024/0[3](https://git.novascotia.ca/digitalplatformservices/test-macos-runner/-/jobs/218302#L3)/12 19:29:40 Pulling the latest version of ghcr.io/cirruslabs/macos-ventura-xcode:latest...
2024/03/12 19:29:40 Cloning and configuring a new VM...
2024/03/12 19:29:40 Waiting for the VM to boot and be SSH-able...
2024/03/12 19:29:52 Was able to SSH!
2024/03/12 19:29:52 VM is ready.
Preparing environment
00:02
Running on admins-Virtual-Machine.local...
Getting source from Git repository
00:05
Fetching changes with git depth set to 20...
Initialized empty Git repository in /Users/admin/builds/digitalplatformservices/test-macos-runner/.git/
Created fresh repository.
Checking out 8c5cf61c as detached HEAD (ref is main)...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:00
WARNING: Starting with version 17.0 the 'build_script' stage will be replaced with 'step_script': https://gitlab.com/groups/gitlab-org/-/epics/6112
$ uname -a
Darwin admins-Virtual-Machine.local 22.6.0 Darwin Kernel Version 22.6.0: Fri Sep 15 13:39:33 PDT 2023; root:xnu-8796.1[4](https://git.novascotia.ca/digitalplatformservices/test-macos-runner/-/jobs/218302#L4)1.3.700.8~1/RELEASE_ARM[6](https://git.novascotia.ca/digitalplatformservices/test-macos-runner/-/jobs/218302#L6)4_VMAPPLE arm64
Cleaning up project directory and file based variables
00:01
Job succeeded