Open marcosgriselli opened 4 months ago
Tracked internally as rdar://130478685
Thank you for this report @marcosgriselli! The example project and data are very helpful. I am investigating this along with @rintaro.
Does this need to move over to swift-syntax or swift?
Does this need to move over to swift-syntax or swift?
At this point I do believe it should be transferred to the swiftlang/swift repo. I just tried doing that though, and it appears that you cannot transfer issues across organization boundaries, and due to the pending apple → swiftlang organization transition we'll need to wait until swift-testing makes the transition. We expect this to happen before too long.
@stmontgomery happy to close and create the issue there unless you want the transfer to keep the issue history clean
We can wait to transfer. The relevant people know about the issue already.
I'm actively looking into this issue:
So far, these patches improves the swift-testing
test compilation performance by 2.6x or so (tested with a file with 500 @Test
/#expect
cases) . I'm exploring other areas for further improvements.
Description
Compiling a large amount of Swift Testing tests is significantly slower than compiling the same amount of tests using XCTest on a clean build.
At 1000 tests in a single file building a test suite with Swift Testing is 24x slower than an almost identical suite using XCTest.
Here's a repo with a project to benchmark: https://github.com/marcosgriselli/swift-testing-performance. A few results from that project:
1000 Tests in a single file:
Performance significantly improves when splitting the tests into different files, still 10x slower than XCTest.
1000 tests split in 10 files:
I'm aware of the macros compilation build time issues outside of swift-syntax and that the team is working on addressing these, is there anything we can do in the meantime to reduce the impact while adopting Swift Testing?
Expected behavior
No response
Actual behavior
No response
Steps to reproduce
Run benchmarks on the shared repo to measure performance.
swift-testing version/commit hash
Xcode 16.0 (16A5171c)
Swift & OS version (output of
swift --version ; uname -a
)swift-driver version: 1.109.2 Apple Swift version 6.0 (swiftlang-6.0.0.3.300 clang-1600.0.20.10) Target: arm64-apple-macosx14.0 Darwin Marcoss-MBP.attlocal.net 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 arm64