search

apple / swift-testing

A modern, expressive testing package for Swift
Apache License 2.0
1.47k stars 52 forks source link

Provide a JSON API to *just* list tests (via the callback, not stdout) #506

Open allevato opened 4 days ago

allevato commented 4 days ago

Description

If the JSON ABI entry point is used with listTests: true in its configuration, then the framework prints the list of all tests to stdout and then continues on executing tests (implementation).

However, a client of the JSON entry point may want to query which tests are available without running anything yet. I think this can be hacked up today by passing in a filter regex that could not possibly match any test ID; then, the framework would terminate after running nothing. (Disclaimer: I haven't actually tried this.) However, doing so would still require the calling process to redirect stdout temporarily and then parse the output line-by-line.

Would it be possible for the JSON entry point to provide a mode where listTests: true (or something like it with a different name, if we need to preserve the current behavior) would report the list of tests back to the caller via the callback as a JSON payload instead, returning immediately without running anything?

Expected behavior

No response

Actual behavior

No response

Steps to reproduce

No response

swift-testing version/commit hash

main

Swift & OS version (output of swift --version && uname -a)

No response

grynspan commented 4 days ago

and then continues on executing tests (implementation).

It exits early, actually.

We have not defined a stable JSON interface for input yet. If and when we do, it will likely include some sort of "list only" flag that produces a sequence of test records but does not run them.

allevato commented 4 days ago

It exits early, actually.

Thanks! You're right—I misread the following else and didn't realize it was surrounding the entirety of the remaining logic.

it will likely include some sort of "list only" flag that produces a sequence of test records but does not run them.

That sounds good to me! I should be able to work with the existing behavior until then.

grynspan commented 3 days ago

Actually, I'm going to reopen this and use it to track an experimental implementation of the described functionality, assuming it won't break VS Code. @plemarquand are you passing any additional arguments when you call swift test list?

stmontgomery commented 3 days ago

Tracked internally with rdar://130627856

plemarquand commented 3 days ago

@grynspan We call it with —skip-build, but otherwise no.