ordo-one / package-benchmark

Swift benchmark runner with many performance metrics and great CI support
Apache License 2.0
313 stars 25 forks source link

SwiftData errors when running Benchmarks. #264

Closed vanvoorden closed 1 month ago

vanvoorden commented 1 month ago
CoreData: error:  Failed to create directory file:///Users/rick/Library/Application%20Support/Benchmarks: NSCocoaErrorDomain (513)
CoreData: fault: Unhandled exception finding default Directory URL '+[NSPersistentContainer defaultDirectoryURL] Could not conjure up a useful location for writing persistent stores.'

https://github.com/vanvoorden/2024-08-26

I am running into these errors when attempting to run a benchmark on a SwiftData context from the 2024-08-26 repo. The errors seem to be harmless… the benchmark continues without crashing on those unhandled exceptions.

https://github.com/vanvoorden/2024-08-02/

I seem to have no problem running against SwiftData from a different Swift Package Executable (2024-08-02). This package seems to build and run with no errors. This package also creates the expected directory under ~/Library/Application%20Support/2024-08-02.

I don't yet completely understand what is happening… but there seems to be some reason why Benchmarks is failing to create that extra directory under Application%20Support.

I can also run Benchmarks with no errors after I add the Application%20Support/Benchmarks directory manually from mac Finder.

The benchmarks seem to be running correctly even after printing those errors… so I am not sure if there is anything important to fix in the Benchmarks package for now.

swift-driver version: 1.115 Apple Swift version 6.0 (swiftlang-6.0.0.9.10 clang-1600.0.26.2)
Target: arm64-apple-macosx14.0
vanvoorden commented 1 month ago

https://developer.apple.com/forums/thread/762847

Also asked on Apple Dev Forums for more details on this Core Data error.

vanvoorden commented 1 month ago
import Benchmark
import CoreData

let benchmarks = {
  Benchmark("Benchmark") { benchmark in
    let _ = NSPersistentContainer.defaultDirectoryURL
  }
}

This crashes:

*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: '+[NSPersistentContainer defaultDirectoryURL] Could not conjure up a useful location for writing persistent stores.'
*** First throw call stack:
(
    0   CoreFoundation                      0x0000000189caa2ec __exceptionPreprocess + 176
    1   libobjc.A.dylib                     0x000000018978e158 objc_exception_throw + 60
    2   CoreData                            0x00000001904d9034 __44+[NSPersistentContainer defaultDirectoryURL]_block_invoke + 0
    3   CoreData                            0x00000001905c3f74 $sSo21NSPersistentContainerC8CoreDataE19defaultDirectoryURL10Foundation0G0VvgZ + 40
    4   Benchmarks                          0x00000001047530e4 $s10Benchmarks10benchmarks9BenchmarkACCSgycvpfiAEycfU_yADcfU_ + 92
    5   Benchmarks                          0x0000000104703608 $s9Benchmark0A8ExecutorV3runySayAA0A6ResultVGA2ACF + 2640
    6   Benchmarks                          0x0000000104728f34 $s9Benchmark0A6RunnerV3runyyYaKFTY0_ + 6768
    7   Benchmarks                          0x0000000104725cb5 $s9Benchmark0A11RunnerHooksPAAE4mainyyYaFZTQ1_ + 1
    8   Benchmarks                          0x00000001047535a5 $sIetH_yts5Error_pIegHrzo_TR10async_MainTf3npf_nTQ0_ + 1
    9   libswift_Concurrency.dylib          0x00000002513d6149 _ZL22completeTaskAndReleasePN5swift12AsyncContextEPNS_10SwiftErrorE + 1
)
libc++abi: terminating due to uncaught exception of type NSException
vanvoorden commented 1 month ago

https://github.com/swiftlang/swift-package-manager/issues/6948

This might be another version of this error from SPM plug ins.

vanvoorden commented 1 month ago
swift package --disable-sandbox benchmark

Passing disable-sandbox seems to silence the errors and I now see the correct Benchmarks directory being created in Application%20Support. This looks like a legit workaround… are there any potential negative side effects to always passing disable-sandbox that we should know more about?

hassila commented 1 month ago

Nah, for all benchmarks that touch disk or network, it's basically a prerequisite on macOS - if you don't want to disable the full sandbox, you can use --allow-writing-to-package-directory (if that is where you write stuff - in the case above, it seems it wanted to write into Application Support, then your only option is to disable the sandbox completely)

https://swiftpackageindex.com/ordo-one/package-benchmark/1.25.0/documentation/benchmark/runningbenchmarks#Network-or-disk-permissions-failures

We disable the sandbox for all our disk/network benchmarks.

vanvoorden commented 1 month ago

@hassila Ok. That makes sense. I'm still a little confused about this error because I explicitly specified this SwiftData ORM context should be in-memory… but it's possible the SwiftData infra just hardcodes some file URL that must be valid at all times no matter what. I am unblocked with the disable-sandbox workaround. Thanks!

hassila commented 1 month ago

Maybe the easiest way to report it would be to try to make an absolute minimal command plugin to SPM - it also uses sandboxing and should then crash with basically apple-only code - suitable for a FB?