search

swiftlang / swift-package-manager

The Package Manager for the Swift Programming Language
Apache License 2.0
9.74k stars 1.34k forks source link

Skipped tests are not reported as skipped in xUnit output #7383

Closed adam-fowler closed 3 months ago

adam-fowler commented 8 months ago

Is it reproducible with SwiftPM command-line tools: swift build, swift test, swift package etc?

Description

The xUnit file output by swift test --parallel --xunit-output <file> does not report when a test is skipped

Expected behavior

xUnit file has an xml node that indicates the test was skipped

Actual behavior

Instead it reports that the test ran as if it passed

Steps to reproduce

1) In SwiftPM project create test with the first line try XCTSkipIf(true) 2) Run swift test --parallel --xunit-output test.xml 3) Inspect test.xml There is no indication in the file that test was skipped

Swift Package Manager version/commit hash

5.9.2

Swift & OS version (output of swift --version ; uname -a)

swift-driver version: 1.87.3 Apple Swift version 5.9.2 (swiftlang-5.9.2.2.56 clang-1500.1.0.2.5) Target: arm64-apple-macosx14.0 Darwin Adams-MBP-M1-Max.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64

grynspan commented 3 months ago

I wish we had a proper JUnit XML schema to follow here. I think this is as close as we have to a spec: https://github.com/junit-team/junit5/blob/main/junit-platform-reporting/src/main/java/org/junit/platform/reporting/legacy/xml/XmlReportWriter.java

It does have "skipped" so we can do that.

grynspan commented 3 months ago

XCTest will need to be taught to report a skipped test in a way Swift Package Manager can detect; that may ultimately need to wait until we teach XCTest how to produce JSON output like we do for Swift Testing, at which point a common implementation is possible.

adam-fowler commented 3 months ago

@plemarquand I don't know if you are following this

grynspan commented 3 months ago

Currently, the XCTest implementation of xUnit (JUnit) XML output is based solely on the exit status of the process hosting each test. Additional information such as the cause of a failure is not available. Resolving this problem will require an overall refactoring of how XCTest produces its XML output. That work is tracked by #4752, so I'm going to mark this issue as a duplicate of that one.

grynspan commented 3 months ago

Duplicate of #4752