Generated tests don't run reliably

dabrahams commented 1 year ago

Description

Check out the test-generation tag from the https://github.com/val-lang/val project, follow the setup instructions if you haven't already, and do swift -vv test. You can clearly see a test file (ValFileTests.swift) being generated, and then being compiled into the EndToEndTests test target. However, the generated XCTestCase is never run. If you open Package.swift in Xcode and run the tests there, however, they run.

There's something flaky going on here, because:

I've tried and failed to reproduce it in the small: https://github.com/dabrahams/TestGeneration works exactly as expected.
I could swear I've seen this work as expected directly in the Val project (verified; see comment below).
There are other weird effects, e.g. changing the source of the test generator plugin doesn't cause it to be re-run.

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.8.0

Swift & OS version (output of `swift --version ; uname -a`)

swift-driver version: 1.75.2 Apple Swift version 5.8 (swiftlang-5.8.0.124.2 clang-1403.0.22.11.100) Target: arm64-apple-macosx13.0 Darwin DaveA-MBP14.luannocracy.com 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64

tomerd commented 1 year ago

cc @neonichu @abertelrud

dabrahams commented 1 year ago

I'm really tearing my hair out over this, so any help you can offer would be much appreciated! I have, from time to time, gotten my system into a state where these tests do run, even with no code changes to the project.

It appears to have something to do with state accumulated in the .build directory, but I know nothing more.

dabrahams commented 1 year ago

With the nightly snapshot, I get this; not sure if it's any help:

xcrun --toolchain swift swift test --filter _val
warning: 'val': found 8 file(s) which are unhandled; explicitly declare them as resources or exclude from the target
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Pragma.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Subscript.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Monomorphize.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Factorial.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Existential.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Lambda.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/Destroy.val
    /Users/dave/src/val/Tests/EndToEndTests/TestCases/LocalInout.val
Building for debugging...
Build complete! (0.43s)
[0/1] Planning build
[1/1] Compiling plugin Format Source Code
[2/2] Compiling plugin GenerateManualPlugin
[3/3] Compiling plugin Lint Source Code
[4/4] Compiling plugin TestGeneratorPlugin
Building for debugging...
[4/8] Generating ValFileTests.swift
[6/8] Compiling EndToEndTests ValFileTests.swift
[7/8] Emitting module EndToEndTests
ld: warning: dylib (/opt/homebrew/Cellar/llvm/16.0.3/lib/libLLVM-16.dylib) was built for newer macOS version (13.0) than being linked (12.0)
ld: warning: dylib (/opt/homebrew/Cellar/llvm/16.0.3/lib/libunwind.dylib) was built for newer macOS version (13.0) than being linked (12.0)
ld: warning: dylib (/opt/homebrew/Cellar/llvm/16.0.3/lib/libunwind.dylib) was built for newer macOS version (13.0) than being linked (12.0)
[7/8] Linking ValPackageTests
Build complete! (6.88s)
error: signalled(6): /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper /Users/dave/src/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest /var/folders/wg/wvvfbgl96f727hzqdtpfrrl80000gn/T/TemporaryFile.u89xwy output:
    dyld[99443]: Library not loaded: @rpath/XCTest.framework/Versions/A/XCTest
      Referenced from: <18FA1015-E205-3FCC-AF71-9555A3EB0095> /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper
      Reason: tried: '/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/libexec/swift/lib/XCTest.framework/Versions/A/XCTest' (no such file), '/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/lib/swift/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/libexec/swift/lib/XCTest.framework/Versions/A/XCTest' (no such file), '/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2023-05-20-a.xctoolchain/usr/lib/swift/macosx/XCTest.framework/Versions/A/XCTest' (no such file), '/System/Library/Frameworks/XCTest.framework/Versions/A/XCTest' (no such file, not in dyld cache)

dabrahams commented 1 year ago

Further report: can't reproduce the problem on my work machine (MacBook Pro M1 Max), but it happens all the time on my home laptop (MacBook Pro M1). 🤔

dabrahams commented 1 year ago

This is amazing. SPM is unhappy with this package in a directory named val unless its package name is also "val". This change makes the problem go away:

diff --git a/Package.swift b/Package.swift
index c92ba74f..1846866b 100644
--- a/Package.swift
+++ b/Package.swift
@@ -7,7 +7,7 @@ let allTargetsSwiftSettings: [SwiftSetting] = [
 ]

 let package = Package(
-  name: "Val",
+  name: "val",

   platforms: [
     .macOS(.v12)

You'd think it would be something like, the directory name needs to match, case-sensitively, but in fact that's not it. I can make the small TestGeneration project fail by renaming its directory val/, but not testgeneration/ or TestGeneration1/, or Val/. On the other hand, renaming the package "val" doesn't fix TestGeneration once its directory has been named val/.

dabrahams commented 1 year ago

NOTE: for reproducibility, all invocations of swift should be prefixed with a swift package clean. Like I said, I've seen it get into a state that appears to be working, but that's gone after I clean.

dabrahams commented 1 year ago

Final note: TestGeneration has now been minimized so that there's nothing inessential. It runs two tests… unless I check it out into a directory named ~/src/val/ on one particular laptop (I've tried this on 4 different macs), in which case it runs zero tests. Checked out into /tmp/val or ~/Downloads/val on that machine, or anywhere else I've tried it, it runs two tests.

dabrahams commented 1 year ago

FWIW, the change cited here no longer seems to suppress the problem. The only answer I currently have is to move the project to a different directory.

dabrahams commented 1 year ago

Moving it to a directory named ~/src/val.0 doesn't suppress the problem, but ~/src/val.1 does.

dabrahams commented 1 year ago

Now the spm-bug-wrinkle tag in the val repository is a state where this command will run the generated tests twice, as it should:

swift package clean && swift test --filter _val && echo '*****************' && swift test --filter _val

…unless the code is in ~/src/val.1 (but not ~/src/val!), in which case, the generated tests only run the second time swift test is invoked by that command.

Is it possible that the determination of which tests need to run is happening concurrently with the test generation, without synchronization?

dabrahams commented 1 year ago

Bump. This bug is also reproduced by @kyouko-taiga and is a serious problem for our project. We are willing to take extraordinary measures to help debug it (e.g. run SPM under the debugger somehow) but we need to know what information you need and we need instructions to get that info.

neonichu commented 1 year ago

Planning to look into this soon.

dabrahams commented 1 year ago

@neonichu Ebullient thanks a billion

neonichu commented 1 year ago

Hm, I only see a single tag on https://github.com/dabrahams/TestGeneration, called missing-unhandled-source-diagnostic-bug but no spm-bug-wrinkle. Am I looking in the wrong place?

dabrahams commented 1 year ago

Yes, that tag is in the (larger) Val repo

dabrahams commented 1 year ago

@neonichu https://github.com/val-lang/val/tree/spm-bug-wrinkle.

BTW:

Is it possible that the determination of which tests need to run is happening concurrently with the test generation, without synchronization?

dabrahams commented 1 year ago

@neonichu how's it going? Is there anything I can do to help you diagnose the problem?

neonichu commented 1 year ago

Sorry haven't had time to try to repro again, but I do see that we're running the entire build before applying the filters here, so I don't think there should be an issue with concurrency.

dabrahams commented 1 year ago

I the scenario I'm imagining, the filters are irrelevant, and I phrased it poorly, so let me try again:

Is it possible that the determination of which tests exist (the unfiltered list) is happening concurrently with the test generation, without synchronization?

neonichu commented 1 year ago

Right, it's basically the same code in that case as well. We can basically only get the list of tests from the final .xctest bundle.

I haven't been able to repro the issue myself so far, but I can also not 100% replicate your setup. It seems like homebrew is only giving me intel binaries, so I have to run swift test through arch -x86_64 to get a working setup which might be different enough from yours to not reproduce the problem maybe

Could you try running just the first swift test invocation that doesn't run any tests and then run the following script:

#!/bin/sh

set -x

sp=`xcrun -f swift-package`
sw=`dirname "$sp"`

"$sw/../libexec/swift/pm/swiftpm-xctest-helper" .build/debug/ValPackageTests.xctest tests.json

That should produce a tests.json which lists the tests we have found through the test helper.

dabrahams commented 1 year ago

As mentioned here, and here, the problem occurs even with the tiny TestGeneration project which has no dependencies, so you shouldn't need anything from homebrew to reproduce this. On the other hand, as also mentioned throughout this issue, we don't really know what is required to reproduce it; it only apparently happens on some machines and when the project is in a directory with a certain names, combined with certain other SPM project names(!)… so I hope I'm still able to observe the problem… who knows what critical condition may have shifted on that machine?

All that said, I will try what you said and get back to you; thank you for your attention!

dabrahams commented 1 year ago

I'm afraid it's no longer reproducing for me directly. I have a snapshot of my Data volume that I took when the issue was happening. If I can figure out how to boot with that as my Data (or if you can tell me), I might be able to get it to happen again, provided it wasn't OS-dependent.

kyouko-taiga commented 1 year ago

The problem still occurs consistently on my setup.

I tried to run the script @neonichu suggested (this one).

Here's the output:

++ xcrun -f swift-package
+ sp=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-package
++ dirname /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-package
+ sw=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
+ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../libexec/swift/pm/swiftpm-xctest-helper .build/debug/ValPackageTests.xctest tests.json
dyld[45953]: Library not loaded: @rpath/XCTest.framework/Versions/A/XCTest
  Referenced from: <7100FADC-D355-3D16-BC03-B672E6BEB0A2> /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/libexec/swift/pm/swiftpm-xctest-helper
  Reason: tried: '/Applications/Xcode.app/SharedFrameworks/XCTest.framework/Versions/A/XCTest' (no such file), '/Applications/Xcode.app/SharedFrameworks/XCTest.framework/Versions/A/XCTest' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/XCTest.framework/Versions/A/XCTest' (no such file), '/Applications/Xcode.app/SharedFrameworks/XCTest.framework/Versions/A/XCTest' (no such file), '/Applications/Xcode.app/SharedFrameworks/XCTest.framework/Versions/A/XCTest' (no such file), '/Library/Frameworks/XCTest.framework/Versions/A/XCTest' (no such file), '/System/Library/Frameworks/XCTest.framework/Versions/A/XCTest' (no such file, not in dyld cache)
exp.sh: line 8: 45953 Abort trap: 6           "$sw/../libexec/swift/pm/swiftpm-xctest-helper" .build/debug/ValPackageTests.xctest tests.json

I guess something went wrong.

neonichu commented 1 year ago

Ah sorry, I think I had these set in my shell env from earlier, so I overlooked them. Could you add these two lines before the swiftpm-xctest-helper line:

export DYLD_FRAMEWORK_PATH=`xcrun --sdk macosx --show-sdk-platform-path`/Developer/Library/Frameworks
export DYLD_LIBRARY_PATH=`xcrun --sdk macosx --show-sdk-platform-path`/Developer/usr/lib

kyouko-taiga commented 1 year ago

I can also reproduce the issue with Dave's https://github.com/dabrahams/TestGeneration if I move the folder around. Results do depend on where I put the repository, seemingly randomly (e.g., moving it from ~/Developer/Temporary/. to ~/Developer/. makes the problem disappear).

As for the script, I got a new error (in the small project):

+ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../libexec/swift/pm/swiftpm-xctest-helper .build/debug/ValPackageTests.xctest tests.json
error: unableToLoadBundle("/Users/alvae/Developer/Temporary/TestGeneration/.build/debug/ValPackageTests.xctest")

neonichu commented 1 year ago

Interesting, that might be a hint on what's going wrong. If we can't even load the bundle, we won't know about the tests inside. I'm surprised that the error isn't bubbled up all the way to the user though.

neonichu commented 1 year ago

Could you please share the output of otool -lL .build/debug/ValPackageTests.xctest/Contents/MacOS/ValPackageTests?

kyouko-taiga commented 1 year ago

There you are:

alvae@navi val % otool -lL  .build/debug/ValPackageTests.xctest/Contents/MacOS/ValPackageTests
.build/debug/ValPackageTests.xctest/Contents/MacOS/ValPackageTests:
Load command 0
      cmd LC_SEGMENT_64
  cmdsize 1832
  segname __TEXT
   vmaddr 0x0000000000000000
   vmsize 0x0000000002788000
  fileoff 0
 filesize 41451520
  maxprot 0x00000005
 initprot 0x00000005
   nsects 22
    flags 0x0
Section
  sectname __text
   segname __TEXT
      addr 0x0000000000002a84
      size 0x00000000020a7610
    offset 10884
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000400
 reserved1 0
 reserved2 0
Section
  sectname __stubs
   segname __TEXT
      addr 0x00000000020aa094
      size 0x000000000000555c
    offset 34250900
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x80000408
 reserved1 0 (index into indirect symbol table)
 reserved2 12 (size of stubs)
Section
  sectname __init_offsets
   segname __TEXT
      addr 0x00000000020af5f0
      size 0x0000000000000570
    offset 34272752
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000016
 reserved1 0
 reserved2 0
Section
  sectname __objc_methlist
   segname __TEXT
      addr 0x00000000020afb60
      size 0x00000000000010a0
    offset 34274144
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_typeref
   segname __TEXT
      addr 0x00000000020b0c00
      size 0x000000000000d1f0
    offset 34278400
     align 2^1 (2)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __cstring
   segname __TEXT
      addr 0x00000000020bddf0
      size 0x000000000010094b
    offset 34332144
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000002
 reserved1 0
 reserved2 0
Section
  sectname __const
   segname __TEXT
      addr 0x00000000021be740
      size 0x0000000000559943
    offset 35383104
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_fieldmd
   segname __TEXT
      addr 0x0000000002718084
      size 0x0000000000009054
    offset 40992900
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_types
   segname __TEXT
      addr 0x00000000027210d8
      size 0x0000000000000b9c
    offset 41029848
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_reflstr
   segname __TEXT
      addr 0x0000000002721c80
      size 0x0000000000004758
    offset 41032832
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_assocty
   segname __TEXT
      addr 0x00000000027263d8
      size 0x00000000000019e8
    offset 41051096
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_proto
   segname __TEXT
      addr 0x0000000002727dc0
      size 0x00000000000024a0
    offset 41057728
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_capture
   segname __TEXT
      addr 0x000000000272a260
      size 0x0000000000001b68
    offset 41067104
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_builtin
   segname __TEXT
      addr 0x000000000272bdc8
      size 0x00000000000002f8
    offset 41074120
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_mpenum
   segname __TEXT
      addr 0x000000000272c0c0
      size 0x0000000000000288
    offset 41074880
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift5_protos
   segname __TEXT
      addr 0x000000000272c348
      size 0x00000000000000f0
    offset 41075528
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __objc_methname
   segname __TEXT
      addr 0x000000000272c438
      size 0x000000000000273a
    offset 41075768
     align 2^0 (1)
    reloff 0
    nreloc 0
     flags 0x00000002
 reserved1 0
 reserved2 0
Section
  sectname __swift5_entry
   segname __TEXT
      addr 0x000000000272eb74
      size 0x0000000000000008
    offset 41085812
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __gcc_except_tab
   segname __TEXT
      addr 0x000000000272eb7c
      size 0x000000000000042c
    offset 41085820
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __ustring
   segname __TEXT
      addr 0x000000000272efa8
      size 0x000000000000010c
    offset 41086888
     align 2^1 (2)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __unwind_info
   segname __TEXT
      addr 0x000000000272f0b4
      size 0x000000000001a2cc
    offset 41087156
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __eh_frame
   segname __TEXT
      addr 0x0000000002749380
      size 0x000000000003ec7c
    offset 41194368
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 1
      cmd LC_SEGMENT_64
  cmdsize 472
  segname __DATA_CONST
   vmaddr 0x0000000002788000
   vmsize 0x000000000015c000
  fileoff 41451520
 filesize 1425408
  maxprot 0x00000003
 initprot 0x00000003
   nsects 5
    flags 0x10
Section
  sectname __got
   segname __DATA_CONST
      addr 0x0000000002788000
      size 0x0000000000007868
    offset 41451520
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000006
 reserved1 1821 (index into indirect symbol table)
 reserved2 0
Section
  sectname __const
   segname __DATA_CONST
      addr 0x000000000278f870
      size 0x0000000000153370
    offset 41482352
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __objc_classlist
   segname __DATA_CONST
      addr 0x00000000028e2be0
      size 0x00000000000000c8
    offset 42871776
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x10000000
 reserved1 0
 reserved2 0
Section
  sectname __objc_catlist
   segname __DATA_CONST
      addr 0x00000000028e2ca8
      size 0x0000000000000008
    offset 42871976
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x10000000
 reserved1 0
 reserved2 0
Section
  sectname __objc_imageinfo
   segname __DATA_CONST
      addr 0x00000000028e2cb0
      size 0x0000000000000008
    offset 42871984
     align 2^2 (4)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Load command 2
      cmd LC_SEGMENT_64
  cmdsize 952
  segname __DATA
   vmaddr 0x00000000028e4000
   vmsize 0x00000000000a8000
  fileoff 42876928
 filesize 49152
  maxprot 0x00000003
 initprot 0x00000003
   nsects 11
    flags 0x0
Section
  sectname __objc_const
   segname __DATA
      addr 0x00000000028e4000
      size 0x0000000000000c20
    offset 42876928
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __objc_selrefs
   segname __DATA
      addr 0x00000000028e4c20
      size 0x0000000000000970
    offset 42880032
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x10000005
 reserved1 0
 reserved2 0
Section
  sectname __objc_classrefs
   segname __DATA
      addr 0x00000000028e5590
      size 0x0000000000000068
    offset 42882448
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x10000000
 reserved1 0
 reserved2 0
Section
  sectname __objc_data
   segname __DATA
      addr 0x00000000028e55f8
      size 0x0000000000000e18
    offset 42882552
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __data
   segname __DATA
      addr 0x00000000028e6410
      size 0x000000000000972a
    offset 42886160
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __s_async_hook
   segname __DATA
      addr 0x00000000028efb40
      size 0x0000000000000190
    offset 42924864
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __swift56_hooks
   segname __DATA
      addr 0x00000000028efcd0
      size 0x00000000000000b0
    offset 42925264
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000000
 reserved1 0
 reserved2 0
Section
  sectname __thread_vars
   segname __DATA
      addr 0x00000000028efd80
      size 0x0000000000000078
    offset 42925440
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000013
 reserved1 0
 reserved2 0
Section
  sectname __thread_bss
   segname __DATA
      addr 0x00000000028efdf8
      size 0x0000000000000028
    offset 0
     align 2^3 (8)
    reloff 0
    nreloc 0
     flags 0x00000012
 reserved1 0
 reserved2 0
Section
  sectname __bss
   segname __DATA
      addr 0x00000000028efe20
      size 0x0000000000094850
    offset 0
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Section
  sectname __common
   segname __DATA
      addr 0x0000000002984670
      size 0x0000000000007800
    offset 0
     align 2^4 (16)
    reloff 0
    nreloc 0
     flags 0x00000001
 reserved1 0
 reserved2 0
Load command 3
      cmd LC_SEGMENT_64
  cmdsize 72
  segname __LINKEDIT
   vmaddr 0x000000000298c000
   vmsize 0x00000000013f8000
  fileoff 42926080
 filesize 20922428
  maxprot 0x00000001
 initprot 0x00000001
   nsects 0
    flags 0x0
Load command 4
      cmd LC_DYLD_CHAINED_FIXUPS
  cmdsize 16
  dataoff 42926080
 datasize 236432
Load command 5
      cmd LC_DYLD_EXPORTS_TRIE
  cmdsize 16
  dataoff 43162512
 datasize 1661536
Load command 6
     cmd LC_SYMTAB
 cmdsize 24
  symoff 44954096
   nsyms 266202
  stroff 49236024
 strsize 14117384
Load command 7
            cmd LC_DYSYMTAB
        cmdsize 80
      ilocalsym 0
      nlocalsym 228033
     iextdefsym 228033
     nextdefsym 36146
      iundefsym 264179
      nundefsym 2023
         tocoff 0
           ntoc 0
      modtaboff 0
        nmodtab 0
   extrefsymoff 0
    nextrefsyms 0
 indirectsymoff 49213328
  nindirectsyms 5674
      extreloff 0
        nextrel 0
      locreloff 0
        nlocrel 0
Load command 8
     cmd LC_UUID
 cmdsize 24
    uuid 7ECAEE3A-4637-3B52-BCE9-DBC60D27698E
Load command 9
      cmd LC_BUILD_VERSION
  cmdsize 32
 platform 1
    minos 13.0
      sdk 13.3
   ntools 1
     tool 3
  version 857.1
Load command 10
      cmd LC_SOURCE_VERSION
  cmdsize 16
  version 0.0
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 1319.100.3
compatibility version 1.0.0
Load command 12
          cmd LC_LOAD_DYLIB
      cmdsize 48
         name /usr/lib/libc++.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 1500.65.0
compatibility version 1.0.0
Load command 13
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /opt/local/lib/libz.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 1.2.13
compatibility version 1.0.0
Load command 14
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /opt/local/lib/libzstd.1.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 1.5.5
compatibility version 1.0.0
Load command 15
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name /opt/local/lib/libncurses.6.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 6.0.0
compatibility version 6.0.0
Load command 16
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /opt/local/lib/libxml2.2.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 13.4.0
compatibility version 13.0.0
Load command 17
          cmd LC_LOAD_DYLIB
      cmdsize 104
         name /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 1971.0.0
compatibility version 150.0.0
Load command 18
          cmd LC_LOAD_DYLIB
      cmdsize 96
         name /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 1971.0.0
compatibility version 300.0.0
Load command 19
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libobjc.A.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 228.0.0
compatibility version 1.0.0
Load command 20
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftAppKit.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 113.0.0
compatibility version 1.0.0
Load command 21
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftCore.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 5.8.0
compatibility version 1.0.0
Load command 22
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 72
         name /usr/lib/swift/libswiftCoreFoundation.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 120.100.0
compatibility version 1.0.0
Load command 23
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftCoreImage.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 2.0.0
compatibility version 1.0.0
Load command 24
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftDarwin.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 0.0.0
compatibility version 1.0.0
Load command 25
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftDispatch.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 20.0.0
compatibility version 1.0.0
Load command 26
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftIOKit.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 1.0.0
compatibility version 1.0.0
Load command 27
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftMetal.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 306.5.16
compatibility version 1.0.0
Load command 28
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftObjectiveC.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 6.0.0
compatibility version 1.0.0
Load command 29
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftQuartzCore.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 3.0.0
compatibility version 1.0.0
Load command 30
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 64
         name /usr/lib/swift/libswiftXPC.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 6.0.0
compatibility version 1.0.0
Load command 31
          cmd LC_LOAD_WEAK_DYLIB
      cmdsize 72
         name /usr/lib/swift/libswift_Concurrency.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 5.8.0
compatibility version 1.0.0
Load command 32
          cmd LC_LOAD_DYLIB
      cmdsize 72
         name @rpath/XCTest.framework/Versions/A/XCTest (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 21772.0.0
compatibility version 0.0.0
Load command 33
          cmd LC_LOAD_DYLIB
      cmdsize 64
         name @rpath/libXCTestSwiftSupport.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969
      current version 21772.0.0
compatibility version 1.0.0
Load command 34
          cmd LC_RPATH
      cmdsize 40
         path @loader_path/../../../ (offset 12)
Load command 35
      cmd LC_FUNCTION_STARTS
  cmdsize 16
  dataoff 44824048
 datasize 130048
Load command 36
      cmd LC_DATA_IN_CODE
  cmdsize 16
  dataoff 44954096
 datasize 0
Load command 37
      cmd LC_CODE_SIGNATURE
  cmdsize 16
  dataoff 63353408
 datasize 495100
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1500.65.0)
    /opt/local/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.13)
    /opt/local/lib/libzstd.1.dylib (compatibility version 1.0.0, current version 1.5.5)
    /opt/local/lib/libncurses.6.dylib (compatibility version 6.0.0, current version 6.0.0)
    /opt/local/lib/libxml2.2.dylib (compatibility version 13.0.0, current version 13.4.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
    /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1971.0.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /usr/lib/swift/libswiftAppKit.dylib (compatibility version 1.0.0, current version 113.0.0, weak)
    /usr/lib/swift/libswiftCore.dylib (compatibility version 1.0.0, current version 5.8.0)
    /usr/lib/swift/libswiftCoreFoundation.dylib (compatibility version 1.0.0, current version 120.100.0, weak)
    /usr/lib/swift/libswiftCoreImage.dylib (compatibility version 1.0.0, current version 2.0.0, weak)
    /usr/lib/swift/libswiftDarwin.dylib (compatibility version 1.0.0, current version 0.0.0)
    /usr/lib/swift/libswiftDispatch.dylib (compatibility version 1.0.0, current version 20.0.0)
    /usr/lib/swift/libswiftIOKit.dylib (compatibility version 1.0.0, current version 1.0.0, weak)
    /usr/lib/swift/libswiftMetal.dylib (compatibility version 1.0.0, current version 306.5.16, weak)
    /usr/lib/swift/libswiftObjectiveC.dylib (compatibility version 1.0.0, current version 6.0.0)
    /usr/lib/swift/libswiftQuartzCore.dylib (compatibility version 1.0.0, current version 3.0.0, weak)
    /usr/lib/swift/libswiftXPC.dylib (compatibility version 1.0.0, current version 6.0.0, weak)
    /usr/lib/swift/libswift_Concurrency.dylib (compatibility version 1.0.0, current version 5.8.0, weak)
    @rpath/XCTest.framework/Versions/A/XCTest (compatibility version 0.0.0, current version 21772.0.0)
    @rpath/libXCTestSwiftSupport.dylib (compatibility version 1.0.0, current version 21772.0.0)

neonichu commented 1 year ago

Hm, nothing jumps out to me here. Unfortunately it seems like swiftpm-xctest-helper doesn't show us the underlying error so it's a bit difficult to determine why the XCTest bundle isn't loadable.

neonichu commented 1 year ago

I guess we can write a small program to load the bundle in the same way:

❯ cat bundle                                  
#!/usr/bin/env swift

import Foundation

let bundlePath = CommandLine.arguments[1]
guard let bundle = Bundle(path: bundlePath) else {
    fatalError("could not load \(bundlePath)")
}
try bundle.loadAndReturnError()

If you run this like ./bundle .build/debug/ValPackageTests.xctest, what's the result?

neonichu commented 1 year ago

Oh you will also need the environment variables from here, otherwise it'll just fail finding XCTest.

kyouko-taiga commented 1 year ago

I got:

Swift/ErrorType.swift:200: Fatal error: Error raised at top level: Error Domain=NSCocoaErrorDomain Code=3588 "dlopen(/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/ValPackageTests, 0x0109): Library not loaded: @rpath/XCTest.framework/Versions/A/XCTest

I guess I must be doing something wrong with the environment variables, but I've got this:

% echo $DYLD_LIBRARY_PATH
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/usr/lib
% echo $DYLD_FRAMEWORK_PATH
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks

neonichu commented 1 year ago

Is there any more output which shows e.g. where the linker looked for XCTest.framework?

kyouko-taiga commented 1 year ago

Here's the full story:

Swift/ErrorType.swift:200: Fatal error: Error raised at top level: Error Domain=NSCocoaErrorDomain Code=3588 "dlopen(/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/ValPackageTests, 0x0109): Library not loaded: @rpath/XCTest.framework/Versions/A/XCTest
  Referenced from: <7ECAEE3A-4637-3B52-BCE9-DBC60D27698E> /Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/ValPackageTests
  Reason: tried: '/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/../../../XCTest.framework/Versions/A/XCTest' (no such file), '/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/../../../XCTest.framework/Versions/A/XCTest' (no such file), '/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file), '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/host/XCTest.framework/Versions/A/XCTest' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/XCTest.framework/Versions/A/XCTest' (no such file), '/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/../../../XCTest.framework/Versions/A/XCTest' (no such file), '/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/../../../XCTest.framework/Versions/A/XCTest' (no such file), '/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file), '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/host/XCTest.framework/Versions/A/XCTest' (no such file), '/Library/Frameworks/XCTest.framework/Versions/A/XCTest' (no such file), '/System/Library/Frameworks/XCTest.framework/Versions/A/XCTest' (no such file, not in dyld cache)" UserInfo={NSLocalizedFailureReason=The bundle couldn’t be loaded., NSLocalizedRecoverySuggestion=Try reinstalling the bundle., NSFilePath=/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/ValPackageTests, NSDebugDescription=dlopen(/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/ValPackageTests, 0x0109): Library not loaded: @rpath/XCTest.framework/Versions/A/XCTest
  Referenced from: <7ECAEE3A-4637-3B52-BCE9-DBC60D27698E> /Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/ValPackageTests
  Reason: tried: '/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/../../../XCTest.framework/Versions/A/XCTest' (no such file), '/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/../../../XCTest.framework/Versions/A/XCTest' (no such file), '/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file), '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/host/XCTest.framework/Versions/A/XCTest' (no such file), '/System/Volumes/Preboot/Cryptexes/OS@rpath/XCTest.framework/Versions/A/XCTest' (no such file), '/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/../../../XCTest.framework/Versions/A/XCTest' (no such file), '/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest/Contents/MacOS/../../../XCTest.framework/Versions/A/XCTest' (no such file), '/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file, not in dyld cache), '/System/Volumes/Preboot/Cryptexes/OS/usr/lib/swift/XCTest.framework/Versions/A/XCTest' (no such file), '/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift/host/XCTest.framework/Versions/A/XCTest' (no such file), '/Library/Frameworks/XCTest.framework/Versions/A/XCTest' (no such file), '/System/Library/Frameworks/XCTest.framework/Versions/A/XCTest' (no such file, not in dyld cache), NSBundlePath=/Users/alvae/Developer/val/.build/arm64-apple-macosx/debug/ValPackageTests.xctest, NSLocalizedDescription=The bundle “ValPackageTests.xctest” couldn’t be loaded.}
Stack dump:
0.  Program arguments: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-frontend -frontend -interpret exp.swift -Xllvm -aarch64-use-tbi -enable-objc-interop -stack-check -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -color-diagnostics -new-driver-path /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-driver -empty-abi-descriptor -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift -module-name exp -disable-clang-spi -target-sdk-version 13.3 -target-sdk-name macosx13.3 -- .build/debug/ValPackageTests.xctest
1.  Apple Swift version 5.8.1 (swiftlang-5.8.0.124.5 clang-1403.0.22.11.100)
2.  Compiling with the current language version
3.  While running user code "exp.swift"
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  swift-frontend           0x00000001082e72a8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  swift-frontend           0x00000001082e628c llvm::sys::RunSignalHandlers() + 112
2  swift-frontend           0x00000001082e78b8 SignalHandler(int) + 344
3  libsystem_platform.dylib 0x00000001807c02a4 _sigtramp + 56
4  libswiftCore.dylib       0x000000018e570218 $ss17_assertionFailure__4file4line5flagss5NeverOs12StaticStringV_SSAHSus6UInt32VtF + 316
5  libswiftCore.dylib       0x000000018e6154f4 $ss5ErrorPsSYRzs17FixedWidthInteger8RawValueSYRpzrlE5_codeSivg + 0
6  libswiftCore.dylib       0x000000010f154208 $ss5ErrorPsSYRzs17FixedWidthInteger8RawValueSYRpzrlE5_codeSivg + 18446744071573859604
7  swift-frontend           0x00000001032bbb10 llvm::orc::runAsMain(int (*)(int, char**), llvm::ArrayRef<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, llvm::Optional<llvm::StringRef>) + 1336
8  swift-frontend           0x000000010321e84c swift::RunImmediately(swift::CompilerInstance&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, swift::IRGenOptions const&, swift::SILOptions const&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >&&) + 10816
9  swift-frontend           0x00000001031dabdc processCommandLineAndRunImmediately(swift::CompilerInstance&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >&&, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, swift::FrontendObserver*, int&) + 492
10 swift-frontend           0x00000001031d564c performCompileStepsPostSILGen(swift::CompilerInstance&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, swift::PrimarySpecificPaths const&, int&, swift::FrontendObserver*) + 1552
11 swift-frontend           0x00000001031d89e8 performCompile(swift::CompilerInstance&, int&, swift::FrontendObserver*) + 3288
12 swift-frontend           0x00000001031d6874 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 4308
13 swift-frontend           0x000000010319b5bc swift::mainEntry(int, char const**) + 4116
14 dyld                     0x0000000180467e50 start + 2544

dabrahams commented 1 year ago

The way I read Boris’ request, he was asking about output that occurred before the crash

kyouko-taiga commented 1 year ago

GitHub should really improve their "reply by email" feature ...

If that was the question, then nothing else was shown.

neonichu commented 1 year ago

Hm odd, it looks like the linker is never consulting the /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks path specified in DYLD_FRAMEWORK_PATH?

kyouko-taiga commented 1 year ago

Seems so. Sorry, I'm really not familiar with dynamic linker configuration, so I'm not sure what I can do to help debug this issue.

neonichu commented 1 year ago

Interesting, that might be a hint on what's going wrong. If we can't even load the bundle, we won't know about the tests inside. I'm surprised that the error isn't bubbled up all the way to the user though.

I don't think this is actually correct. When I deliberately break the DYLD_FRAMEWORK_PATH calculation in swift test itself, I do see the dyld loading error bubbled up all the way to the user. So it doesn't seem as if this can be the issue we're actually chasing here.

dabrahams commented 1 year ago

What are the next steps here? I worry that we'll lose the one reliable reproducer we have, which is currently on @kyouko-taiga's machine, before we solve it.

neonichu commented 1 year ago

Not sure tbh. It seems like generally this works, also evidenced by it being hard to reproduce, so my best guess is that something is going wrong with test discovery / swiftpm-xctest-helper but I don't know enough about that so have any concrete ideas on where to look further. Possibly @stmontgomery has any clues?

stmontgomery commented 1 year ago

Nothing jumps out to me, either. The small bundle script reproducer above seems helpful for narrowing down the culprit, though. Can you re-run that, but first set this DYLD env var:

export DYLD_PRINT_SEARCHING=1

And then collect that output? It should spew a lot of logging about all the various paths the dynamic linker attempts to locate/load every library from, and we need to see where it's looking for XCTest.

kyouko-taiga commented 1 year ago

Here's what I got:

Swift/ContiguousArrayBuffer.swift:600: Fatal error: Index out of range
Stack dump:
0.  Program arguments: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-frontend -frontend -interpret exp.swift -Xllvm -aarch64-use-tbi -enable-objc-interop -stack-check -sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX13.3.sdk -color-diagnostics -new-driver-path /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/swift-driver -empty-abi-descriptor -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/swift -module-name exp -disable-clang-spi -target-sdk-version 13.3 -target-sdk-name macosx13.3
1.  Apple Swift version 5.8.1 (swiftlang-5.8.0.124.5 clang-1403.0.22.11.100)
2.  Compiling with the current language version
3.  While running user code "exp.swift"
Stack dump without symbol names (ensure you have llvm-symbolizer in your PATH or set the environment var `LLVM_SYMBOLIZER_PATH` to point to it):
0  swift-frontend           0x0000000107cc72a8 llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) + 56
1  swift-frontend           0x0000000107cc628c llvm::sys::RunSignalHandlers() + 112
2  swift-frontend           0x0000000107cc78b8 SignalHandler(int) + 344
3  libsystem_platform.dylib 0x00000001807c02a4 _sigtramp + 56
4  libsystem_platform.dylib 0xffff80018e570fa0 _sigtramp + 18446603336453655860
5  libswiftCore.dylib       0x000000018e570ce4 $ss17_assertionFailure__4file4line5flagss5NeverOs12StaticStringV_A2HSus6UInt32VtFySRys5UInt8VGXEfU_yAMXEfU_ + 200
6  libswiftCore.dylib       0x000000018e570adc $ss17_assertionFailure__4file4line5flagss5NeverOs12StaticStringV_A2HSus6UInt32VtFySRys5UInt8VGXEfU_ + 212
7  libswiftCore.dylib       0x000000018e57063c $ss17_assertionFailure__4file4line5flagss5NeverOs12StaticStringV_A2HSus6UInt32VtF + 236
8  libswiftCore.dylib       0x000000018e54bda8 $ss22_ContiguousArrayBufferV8capacitySivg + 0
9  libswiftCore.dylib       0x000000018e5505e4 $sSayxSicig + 88
10 libswiftCore.dylib       0x000000010eb34048 $sSayxSicig + 18446744071568243388
11 swift-frontend           0x0000000102c9bb10 llvm::orc::runAsMain(int (*)(int, char**), llvm::ArrayRef<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > >, llvm::Optional<llvm::StringRef>) + 1336
12 swift-frontend           0x0000000102bfe84c swift::RunImmediately(swift::CompilerInstance&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&, swift::IRGenOptions const&, swift::SILOptions const&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >&&) + 10816
13 swift-frontend           0x0000000102bbabdc processCommandLineAndRunImmediately(swift::CompilerInstance&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >&&, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, swift::FrontendObserver*, int&) + 492
14 swift-frontend           0x0000000102bb564c performCompileStepsPostSILGen(swift::CompilerInstance&, std::__1::unique_ptr<swift::SILModule, std::__1::default_delete<swift::SILModule> >, llvm::PointerUnion<swift::ModuleDecl*, swift::SourceFile*>, swift::PrimarySpecificPaths const&, int&, swift::FrontendObserver*) + 1552
15 swift-frontend           0x0000000102bb89e8 performCompile(swift::CompilerInstance&, int&, swift::FrontendObserver*) + 3288
16 swift-frontend           0x0000000102bb6874 swift::performFrontend(llvm::ArrayRef<char const*>, char const*, void*, swift::FrontendObserver*) + 4308
17 swift-frontend           0x0000000102b7b5bc swift::mainEntry(int, char const**) + 4116
18 dyld                     0x0000000180467e50 start + 2544
zsh: trace trap  ./exp.swift

stmontgomery commented 1 year ago

It looks like the DYLD_PRINT_SEARCHING env var didn’t take effect for some reason, I would expect to see lots of dyld output to stdout/err before this

kyouko-taiga commented 1 year ago

I double checked that my environment was configured as directed. Is there anything else I can do to get you a better trace?

neonichu commented 1 year ago

One thing I am separately wondering about: I did discover we had a version of the XCTest helper in SwiftPM's source tree while investigating this: https://github.com/apple/swift-package-manager/pull/6667

In the case of swift.org toolchains, that would be the helper tool being used, so I am wondering if there's any link to the issue here. Could it be that the issue happens in the case of using a swift.org toolchain, but not when using the toolchain from an Xcode installation?

dabrahams commented 1 year ago

@neonichu I'm not sure who you're asking, but if it's me or @kyouko-taiga, I imagine you'll need to request specific information about our systems, because we probably have no idea how to answer. One thing I can say: homebrew installation adds the command-line tools even when there's an Xcode on the system, which seems to cause some problems… but I am the only one of the two of us using homebrew.

neonichu commented 1 year ago

Not sure how to be more specific, I am asking where does the swift come from where this issue is being encountered. Are you (either one of you, but I think only @kyouko-taiga can reproduce the issue at this time if I understand correctly) using the one from Xcode or the one from a toolchain you downloaded from swift.org? SwiftPM in the Commandline Tools doesn't work AFAIK, so we can probably rule that out, but it should also be identical to the one from Xcode as part as the swiftpm-xctest-helper is concerned.

kyouko-taiga commented 1 year ago

I'm using Swift from Xcode and have installed no toolchain from swift.org.

neonichu commented 1 year ago

Thanks, so I think we can rule out the idea of a stale swiftpm-xctest-helper being the issue.

swiftlang / swift-package-manager