search

swiftlang / swift-package-manager

The Package Manager for the Swift Programming Language
Apache License 2.0
9.74k stars 1.34k forks source link

Generated tests don't run reliably #6595

Open dabrahams opened 1 year ago

dabrahams commented 1 year ago

Description

Check out the test-generation tag from the https://github.com/val-lang/val project, follow the setup instructions if you haven't already, and do swift -vv test. You can clearly see a test file (ValFileTests.swift) being generated, and then being compiled into the EndToEndTests test target. However, the generated XCTestCase is never run. If you open Package.swift in Xcode and run the tests there, however, they run.

There's something flaky going on here, because:

  1. I've tried and failed to reproduce it in the small: https://github.com/dabrahams/TestGeneration works exactly as expected.
  2. I could swear I've seen this work as expected directly in the Val project (verified; see comment below).
  3. There are other weird effects, e.g. changing the source of the test generator plugin doesn't cause it to be re-run.

Swift Package Manager version/commit hash

Swift Package Manager - Swift 5.8.0

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.75.2 Apple Swift version 5.8 (swiftlang-5.8.0.124.2 clang-1403.0.22.11.100) Target: arm64-apple-macosx13.0 Darwin DaveA-MBP14.luannocracy.com 22.4.0 Darwin Kernel Version 22.4.0: Mon Mar 6 20:59:28 PST 2023; root:xnu-8796.101.5~3/RELEASE_ARM64_T6000 arm64