Invoking custom test runners via the (new parallel) build system

[dweiller]

Since the parallel build system changes got merged, custom test runners do not interact properly with the build system due to assumptions made in lib/std/Build.zig about how test runners behave. Currently the main way of running tests via the build system (std.Build.addRunArtifact()) assumes that test executables support the client-server API of the standard test runner. I think that the build system should not try to use the client-server API for custom test runners unless the user indicates that the test runner supports it via some new optional field on std.Build.CompileStep.

I'm not sure I see a strong reason to restrict what custom test runners can do (though if others disagree I'm interested to hear and discuss) by enforcing that they follow the client-server communication protocol of the standard test runner. In particular at the moment it prevents test runners from freely using stderr and stdout (the build runner will consume the stdout and report everything written to stderr as an error while still reporting successful execution in the build summary).

[matu3ba]

I'm not sure I see a strong reason to restrict what custom test runners can do

Agreed. The build system should not affect the test runner except where necessary.

In particular at the moment it prevents test runners from freely using stderr and stdout

Afaiu, my PR for "portable pipes" / extra/non-standard streams #14152 would enable changing the build system to not use stdout or stderr, but Andrew wanted to move child_process.zig into process.zig, so it is unlikely to be merged before that.

I think the PR is reasonable to enable custom testrunner behavior for the meantime.

Update: I changed my mind. As bad as only having stdin, stdout and stderr may be, they dont leak file descriptors all over the place (unfixable on unix) or is forced to do a full-stop mutex for all process spawning calls.