Started working on #4 (support for stable Rust). First issue we need to solve is to get access to the harness (since we don't really want to implement it ourselves).
There is https://crates.io/crates/libtest crate, which is recent version of Rust internal test harness, extracted as a crate. However, it only compiles on nightly, so it won't help us here.
There is also https://crates.io/crates/rustc-test, but it is 2 years old. I haven't checked its features, but might not support some of the desired functionality (like, JSON output in tests? colored output?).
So, the third option (which I'm using here) is to use test crate from the Rust itself and also set RUSTC_BOOTSTRAP=1 for our crate so we can access it on stable channel. Not great, but works for now.
Second issue is to get access to the tests. On nightly, we use #[test_case] to hijack Rust tests registration so we can get access to them in nightly.
The caveat here is potentially hitting https://github.com/dtolnay/inventory/issues/7 issue which would manifest itself as test being silently ignored. Not great, but let's see how bad it will be.
Third piece of the puzzle is to intercept execution of tests. This is done by asking users to use harness = false in their Cargo.toml, in which case we take full control of test execution.
So, the plan here is to provide #[datatest::test] attribute that will behave similar to built-in #[test] attribute, but will use our own registry for tests. No need to support #[bench] as it is not supported on stable channel anyway.
The caveat in this case is that if you use built-in #[test], your test will be silently ignored. Not great, not sure what to do about it.
Started working on #4 (support for stable Rust). First issue we need to solve is to get access to the harness (since we don't really want to implement it ourselves).
There is https://crates.io/crates/libtest crate, which is recent version of Rust internal test harness, extracted as a crate. However, it only compiles on nightly, so it won't help us here.
There is also https://crates.io/crates/rustc-test, but it is 2 years old. I haven't checked its features, but might not support some of the desired functionality (like, JSON output in tests? colored output?).
So, the third option (which I'm using here) is to use
testcrate from the Rust itself and also setRUSTC_BOOTSTRAP=1for our crate so we can access it on stable channel. Not great, but works for now.Second issue is to get access to the tests. On nightly, we use
#[test_case]to hijack Rust tests registration so we can get access to them in nightly.Cannot do that on stable. What would help here is something along the lines of https://internals.rust-lang.org/t/idea-global-static-variables-extendable-at-compile-time/9879 or https://internals.rust-lang.org/t/pre-rfc-add-language-support-for-global-constructor-functions. Don't have that, so we use https://crates.io/crates/ctor crate to build our own registry of tests, similar to https://crates.io/crates/inventory.
The caveat here is potentially hitting https://github.com/dtolnay/inventory/issues/7 issue which would manifest itself as test being silently ignored. Not great, but let's see how bad it will be.
Third piece of the puzzle is to intercept execution of tests. This is done by asking users to use
harness = falsein theirCargo.toml, in which case we take full control of test execution.Finally, the last challenge is that with
harness = false, we don't have a good way to intercept "standard" tests (#[test]): https://users.rust-lang.org/t/capturing-test-when-harness-false-in-cargo-toml/28115So, the plan here is to provide
#[datatest::test]attribute that will behave similar to built-in#[test]attribute, but will use our own registry for tests. No need to support#[bench]as it is not supported on stable channel anyway.The caveat in this case is that if you use built-in
#[test], your test will be silently ignored. Not great, not sure what to do about it.Proper solution, of course, would be driving RFC for custom test frameworks: https://github.com/rust-lang/rust/issues/50297 😅
Partially fixes #4 (still missing support for standard tests and also documentation).