Currently, if the swift build or swift test commands fail, the GH action step will still be marked as succeeded.
This seems to be an artefact of the way we were re-running the tests from within the Godot engine; it resulted in the overall failure count being 0 even if there were actual failures.
This PR changes the way we run the tests, and should result in a non-zero exit status if a test does fail.
It also separates the XCTest run and the Swift testing run into two steps.
The default behaviour of swift test is to first run the XCTest engine, then run Swift testing engine.
This results in the following confusing output at the end of the test run:
Test run started.
Testing Library Version: 102 (arm64e-apple-macos13.0)
Test run with 0 tests passed after 0.001 seconds.
This is the output from the Swift testing engine. Because all our tests are using XCTest, and we have no Swift testing tests, it is saying that 0 tests passed. As it's the last thing in the report, it's a bit confusing and for a while I thought that this was why the exit status was 0.
To avoid this confusion, we can explicitly choose a test ending by passing --disable-swift-testing or --disable-xctest.
I have done this in two separate steps, so that the output from each testing engine is clearer.
The Swift testing engine step is a null-op right now, so we could remove it, but at some point we might want to migrate some tests to the new testing framework, so I figured it was worth having it there to avoid later confusion.
migueldeicaza
commented
3 weeks ago
Currently, if the
swift buildorswift testcommands fail, the GH action step will still be marked as succeeded.This seems to be an artefact of the way we were re-running the tests from within the Godot engine; it resulted in the overall failure count being 0 even if there were actual failures.
This PR changes the way we run the tests, and should result in a non-zero exit status if a test does fail.
It also separates the XCTest run and the Swift testing run into two steps.
The default behaviour of
swift testis to first run the XCTest engine, then run Swift testing engine.This results in the following confusing output at the end of the test run:
This is the output from the Swift testing engine. Because all our tests are using XCTest, and we have no Swift testing tests, it is saying that 0 tests passed. As it's the last thing in the report, it's a bit confusing and for a while I thought that this was why the exit status was 0.
To avoid this confusion, we can explicitly choose a test ending by passing
--disable-swift-testingor--disable-xctest.I have done this in two separate steps, so that the output from each testing engine is clearer.
The Swift testing engine step is a null-op right now, so we could remove it, but at some point we might want to migrate some tests to the new testing framework, so I figured it was worth having it there to avoid later confusion.