search

swiftlang / swift-corelibs-xctest

The XCTest Project, A Swift core library for providing unit test support
swift.org
Apache License 2.0
1.15k stars 267 forks source link

How do I get a list of the tests? #488

Closed dabrahams closed 6 months ago

dabrahams commented 6 months ago

I see in XCTestMain.swift there's clearly support for a -l option, but it's unclear what executable I'm supposed to invoke with that option. I need to get a list so that I can parallelize tests. Also, is there a corresponding command that will work on MacOS with Apple's proprietary XCTest?

Many thanks in advance!

grynspan commented 6 months ago

Try swift test list.

dabrahams commented 6 months ago

Sorry, I'm not using SPM. I can try to crawl its code but was hoping you'd tell me how it's done.

grynspan commented 6 months ago

On Darwin, objc_copyClassList() and other Objective-C runtime functions are used to find all subclasses of XCTestCase and their methods starting with test.

On non-Darwin platforms, there is no Objective-C API and Swift does not maintain metadata for individual member functions. At compile time, a list of tests is generated by SourceKit-LSP and/or IndexStore and emitted into the build product's main function by Swift Package Manager.

dabrahams commented 6 months ago

On Darwin, objc_copyClassList() and other Objective-C runtime functions are used to find all subclasses of XCTestCase and their methods starting with test.

Sure, I knew that, but the details are missing for me: is the test binary loaded dynamically, either by SPM or by the XCTest executable? If the latter, presumably there's a way to invoke XCTest to list the tests, so that SPM can drive it? I observed that when testing is parallel, many copies of the XCTest process are running, each with a specific test name—I've assumed that SPM is invoking that executable in parallel somehow, but maybe it's done differently… those are the details I'm looking for on macOS.

On non-Darwin platforms, there is no Objective-C API and Swift does not maintain metadata for individual member functions. At compile time, a list of tests is generated by SourceKit-LSP and/or IndexStore and emitted into the build product's main function by Swift Package Manager.

Sure, I know that too (and have my own tool to do that for other build systems)! The question isn't how the test names get into the binary artifact, but how SPM gets them back out so it can invoke each one in a separate process.

Thanks

briancroom commented 6 months ago

This is very much a part of the system where the source is the documentation, and it's all subject to change without notice, but here are a few quick notes about how things work today:

is the test binary loaded dynamically, either by SPM or by the XCTest executable? If the latter, presumably there's a way to invoke XCTest to list the tests, so that SPM can drive it?

For macOS, there's a program inside of Xcode.app called swiftpm-xctest-helper which dynamically loads the test bundle and emits a JSON structure representing the suites and tests within the bundle.

For other platforms which use swift-corelibs-xctest, SwiftPM runs the built test executable with a --dump-tests-json argument, which instructs it to emit an analogous JSON structure.

A couple of primary source files of interest for these behaviors are: https://github.com/apple/swift-package-manager/blob/main/Sources/Commands/Utilities/TestingSupport.swift https://github.com/apple/swift-corelibs-xctest/blob/main/Sources/XCTest/Private/TestListing.swift