kazk
commented
4 years ago Yeah, I've been waiting for them to add at least JSON output.
It's currently transformed with the following functions:
// Until Rust adds machine readable test output or make formatter availale [1],
// we have to parse the human readable one.
// [1]: https://internals.rust-lang.org/t/past-present-and-future-for-rust-testing/6354
// [2]: https://github.com/rust-lang/rfcs/blob/master/text/2318-custom-test-frameworks.md
// [3]: https://blog.jrenner.net/rust/testing/2018/08/06/custom-test-framework-prop.html
function transformStdout(stdout: string) {
const ss = stdout.split(/\n((?:---- \S+ stdout ----)|failures:)\n/);
// Save test failures in object to use in output.
// There shouldn't have name collisions since test cases are rust functions.
const failures: { [k: string]: string[] } = {};
for (let i = 1; i < ss.length; ++i) {
const s = ss[i];
const m = s.match(/^---- (\S+) stdout ----$/);
if (m === null) continue;
if (i + 1 === ss.length) break;
const name = m[1];
const fail: string[] = [];
// Rust versions before 1.33 had `\t` prepended
const x = ss[++i].replace(/^\t/, "");
const m2 = x.match(/thread '[^']+' panicked at '([^]+)'.*/);
if (m2 === null) continue;
if (m2.index !== 0) fail.push(x.slice(0, m2.index)); // user logged output
fail.push(
`\n<FAILED::>${m2[1].replace(/\\'/g, "'").replace(/\n/g, "<:LF:>")}`
);
failures[name] = fail;
}
const out: string[] = [];
for (const s of ss[0].split("\n")) {
const m = s.match(/^test (\S+) \.{3} (FAILED|ok)$/);
if (m === null) continue;
out.push(`\n<IT::>${m[1].replace(/^tests::/, "")}`);
if (m[2] === "ok") {
out.push("\n<PASSED::>Test Passed");
} else {
out.push.apply(out, failures[m[1]]);
}
out.push(`\n<COMPLETEDIN::>`);
}
out.push("");
return out.join("\n");
}function transformStderr(stderr: string) {
// remove logs from cargo test
stderr = stderr
.replace(/^\s+Compiling .*$/gm, "")
.replace(/^\s+Finished .*$/m, "")
.replace(/^\s+Running .*$/m, "")
.replace(/^\s+Doc-tests .*$/m, "")
.replace("To learn more, run the command again with --verbose.", "")
.replace(/^error: test failed$/m, "")
.replace(/^error: test failed, to rerun pass '--lib'$/m, "")
.replace(/^error: Could not compile .*$/m, "")
.trim();
if (stderr !== "") stderr += "\n";
if (/^error: /m.test(stderr)) {
stderr +=
"\n\nNOTE: Line numbers in error messages can be incorrect due to concatenation.\n";
}
return stderr;
}stdout and stderr are just the raw outputs from cargo test --lib.
mjpieters
Currently, the Rust runner (rightfully) uses the built-in
cargo testtest runner. It's the defacto framework for Rust tests.However, that framework also has its limitations, in its rigidity. Sometimes, you just need to control the output format and order, and that's not something that can be done with the Rust runner alone (it runs tests in parallel, and doesn't offer any way of controlling test order or test skipping).
By default, the runner captures any output produced with the
println!()andeprintln!()macros (to display this output for failed tests, normally). But content written to a newstd::io::stdout()handle is not capured, so a Rust test could still output the Codewars tagged test progress output, simply by not using the[e]println!()macros.Unfortunately, the current CodeWars Rust runner doesn't support this use-case. While I understand that this repository is no longer reflecting how the CodeWars runners actually work, I'm assuming that it's not too far removed from the
rust.jsrunner still visible here; that runner discards all output other than the standard Rust test success and failure text.Please support forwarding any tagged output on stdout, so we can produce richer test suites in Rust.
The use case
Why do I want this? Because I want to provide a Rust translation for the BECOME IMMORTAL kata. The test suite for this Kata tells a story. It sets time limits. It executes tests in phases, with colour-coded sections (
<:LF:>breaks with added newline and indentation for readability):And we can't provide this in a Rust translation of the Kata because there are no options to produce the same kind of output.
If the option existed, I'd write one Rust test case, which then uses a few Rust macros to provide the needed test framework support, so we can produce sequential output.
Offer to do it myself
I'd be happy to produce a diff or pull request, if there was a way to access the docker and runner setup somewhere.
It's not much work, provided the macros take care to lock the handle and start output with a newline to avoid interleaving with the
test [module]::[testfunc] ...andok/FAILEDoutput. Of course, the test runner could always be configured to run single-threaded withcargo test -- --test-threads=1...Sample test case
Here is a sample test snippet that only outputs one of the documentation examples (directly as raw bytes for simplicity):