JSON output for Rust `libtest` require nightly since 1.70

[kazk]

Supporting 1.70 requires extra work/research because -Zunstable-options --format json we've been using to get JSON output is no longer accepted on stable release.

I won't use the nightly because the maintenance cost on each version bump will increase (nightly features will be used and will break). A workaround mentioned is to set RUSTC_BOOTSTRAP=1 environment variable, but that's basically the same as using nightly.

--format json won't be stabilized anytime soon.

Options

1. Use a different test runner with a machine readable output.
   • cargo-nextest supports JUnit XML. We should be able to convert this to Codewars messages. But the file is created after the test, so partial results are lost if the test crashes or timeouts.
   • cargo-nextest has partial support JSON output since 0.9.65.
     • https://nexte.st/book/run-machine-readable
     • Tracking issue https://github.com/nextest-rs/nextest/issues/1152
2. Parse the text output. We used to do this long time ago.
3. Support Rust 1.69 and wait for a bit.
4. Use nightly, but make it behave like stable by disabling all unstable features? (-Zallow-features= (allow nothing))

---

Please comment if you have other ideas.

[hobovsky]

I run nextest locally and here are some findings:

• the JSON output works in general, and can be postprocessed to build a test report. One thing which would have to be verified is handling of logs coming from user solution and tests, especially of runner markup like <LOG::>, <TAB::>, or failure reason if we wanted to wrap it with <ERROR::> etc.
• The JSON output is produced after tests end. You cannot consume it line by line, or result by result, you need to wait for a test suite to complete, with all its consequences like no partial results on a hard crash or timeout. This part makes the JSON format on par with JUnit XML, which we already have postprocessors for (I think we do?).
• By using --no-capture, you can get a stream of results as they come. There are problems with it tho:
  • --no-capture produces a human readable, tap-like output (or is it TAP?), not JSON, and it needs to be parsed,
  • with --no-capture, output of some macros still does not end up on stdout. It gets println!, but for example panic! or dbg! is visible on console screen, but not in redirected stdout (panic! and dbg! are presented in JSON as expected).

[awesomeAD1]

Could you explain the possible downsides, if any, to using nightly with -Zallow-features= (empty list)? The way I understand it, this would result in exactly the same behaviour as stable (unstable features are errors, even when the user users the attribute macro to enable them). This should be a transparent setting as far as the user is concerned, since the runner doesn't expose details about the toolchain it's running on (it could be a undocumented, if not exactly secret, configuration detail). Then, whenever the JSON stuff gets in to stable, just switch back without any breakage.

[awesomeAD1]

In an attempt to answer my own question, I've done some googling and reading. It appears the only "major" difference between nightly (without feature flags) and stable is the more extensive testing plus the six weeks of time to discover bugs, and the slight chance of having compiler bugs. However, any fixes are backported to the current nightly. So choosing a nightly release relatively close to the end of a beta period should offer relative safety in that regard. The only other issue might be the occasional regression in the JSON feature itself, since it is actively WIP. However, the comparatively infrequent updates to the Codewars Rust version (every other, or even third, release), coupled with the fact that any such regressions would be contained to the Runner itself (as opposed to being user facing) means there would be ample opportunity to apply whatever quick fixes necessary on the server side when we do update to a new nightly version. After all, it's not like we'd have to test the version against all Rust kata in the database, is it? A change in the JSON output would be reflected in the output of any kata, so a single test would suffice. And from my understanding, compiler bugs that sneak in tend to evince themselves in very niche circumstances, meaning they'd affect a very small subset of existing solutions, if any.

Unless someone can point out something egregious I've overlooked, this seems to me the best way forward so we can finally stop stagnating on 1.66 forever (it's been more than a year now?) :)