Test case macro should not rely on implementation details of the built-in test framework

[mzabaluev]

It seems the test_case attribute can only work as a hack of the built-in test framework rather than an "honest" attribute macro. In particular, this fails:

use test_case::test_case as parametric_test;

#[parametric_test(1 ; "one")]
#[parametric_test(2 ; "two")]
#[parametric_test(3 ; "three")]
fn test_blubbi(x: u32) {
}

The built-in use of test_case is not documented and should probably be treated as unstable. The built-in attribute certainly is unstable, so overriding it with use test_case::test_case is a hack that is liable to stop working in any future release of Rust. Also, this does not work as intended:

use test_case::test_case;
mod foo {
    use super::*;

    #[test_case(1 ; "one")]
    #[test_case(2 ; "two")]
    #[test_case(3 ; "three")]
    fn test_blubbi(x: u32) {
        // test logic that uses x: {1,2,3} * T: {Foo, Bar} thus 6 test cases in total
    }
}

This impedes interoperability with another boilerplate-reducing solution for tests: https://github.com/mzabaluev/generic-tests/issues/6

[frondeus]

First of all, thank you for creating this issue.

We are not using the built-in use of test_case. I will debug it today why the parametric_test is not working because it definitely should.

I don't understand your last example though, what is expected behaviour in it?

[mzabaluev]

I don't understand your last example though, what is expected behaviour in it?

The compilation fails with:

error[E0659]: test_case is ambiguous

This is because in the foo module, there are now two glob imports, one from use super::*; and the other from the implicit language prelude that also has test_case.

[luke-biel]

Okay, so let me give you some background. test-case came out way before built-in test_case macro was introduced, in Aug 20, 2017 to be precise (though it was a different repository). core::test_case was committed to rust core library, as I see, Sep 5, 2018. At that time we didn't really care for the syntax breakage, switched from using extern imports to current use test_case::test_case and had tests working again.

Test case works by grouping all cases first and then emitting a module with them all. This let's us use rust's built-in test harness and still display a nice hierarchy, with all cases grouped into a parent with test name.

Unfortunately we cannot emit multiple modules with the same name:

3 | mod testing {}
  | ----------- previous definition of the module `testing` here
4 |
5 | mod testing {}
  | ^^^^^^^^^^^ `testing` redefined here
  |
  = note: `testing` must be defined only once in the type namespace of this module

so this would call for a different solution. Rust, without custom test harness, does not let you to re-define test method "display" after you call cargo test, thus there's no way to overcome this.

So tldr - we are open to suggestions and contributions regarding the topic on how to better structure test_case in such way that it outputs useful information to the commandline, while making it more robust, but we ourselves are fine with how it is right now.

PS: There's also another option, because with test-case-core you could easily create a similar crate but with functionalities you need.