[SR-8436] xctest should have an option to emit parsable output

[aciidgh]

Previous ID	SR-8436
Radar	rdar://problem/42828674
Original Reporter	@aciidb0mb3r
Type	Bug

Additional Detail from JIRA

| | | |------------------|-----------------| |Votes | 1 | |Component/s | XCTest | |Labels | Bug | |Assignee | None | |Priority | Medium | md5: 5c1c8b6daff9759845d17ee54c42f4a4

Issue Description:

Similar to the Swift compiler, xctest should provide a way to emit parsable output so the clients can control and format the results.

[aciidgh]

@swift-ci create

[keith]

If you have thoughts on the approach here I might be interested in implementing this.

[aciidgh]

That would be awesome! Have you seen Swift's parsable output? It would be great if we can make something similar in XCTest.

/cc @briancroom @ddunbar

[keith]

Yes I have. So we'd be willing to entirely transform the output coming from Darwin's XCTest?

[aciidgh]

We basically need two implementations here:

1. corelibs-xctest for Linux. This should be somewhat straightforward. We can add a -parseable-output option to the test binaries.

2. I am not clear on what should happen for Darwin. SwiftPM uses the xctest command-line tool to run the tests. A -parseable-output option on xctest command-line tool would make sense to me but that tool is not open-sourced. It looks like we can use XCTestObservation APIs to generate the required output but that would mean we can't use the xctest tool.

Lets wait for @briancroom's thoughts on this.

[briancroom]

This is indeed a long-standing request for Darwin XCTest as well. If we can come up with a format that we're comfortable with, I think we could provide an implementation there as well, via an opt-in command line argument.

I assume you would like this to be output in a streaming fashion as the tests are run?

One complication to any attempt at parsing test results via stdout/stderr is that, since arbitrary user code is running in the process, XCTest's output can end up interleaved with output from the tests themselves. There's not really anything we can do about that besides moving to a model of using a separate pipe for result reporting, which is what Xcode does when running tests through the tool. The structured format being discussed here could potentially be fed either to stderr or to an alternate sink (e.g. a named pipe) to provide a solution here.

[aciidgh]

Yep, a streaming output on stderr/separate pipe sounds perfect!

[MaxDesiatov]

In the meantime, here's a PR that at least allows making test observations (i.e. reports) configurable in situations where you can call XCTMain directly (i.e. LinuxMain.swift) https://github.com/apple/swift-corelibs-xctest/pull/306