Fails under cargo nextest

[wmmc88]

Cargo Nextest has a different test execution model where each test is compiled into a separate binary. This causes issues with macrotest- it seems that it assumes that each crate will contain at most one call to macrotest from a single binary.

This is reproducible from the tests within this repo:

cargo install cargo-nextest
git clone https://github.com/eupn/macrotest.git
cd macrotest/test-project
cargo nextest run

Output:

cargo nextest run
    Finished test [unoptimized + debuginfo] target(s) in 0.04s
    Starting 9 tests across 3 binaries (run ID: 0c2815c1-f154-41a9-bf6f-819b59413968, nextest profile: default)
        FAIL [   0.194s] test-project::par_tests_regression parallel_2

--- STDOUT:              test-project::par_tests_regression parallel_2 ---

running 1 test
Running 1 macro expansion tests
test parallel_2 ... FAILED

failures:

failures:
    parallel_2

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 1 filtered out; finished in 0.07s

--- STDERR:              test-project::par_tests_regression parallel_2 ---
thread 'parallel_2' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

   Canceling due to test failure: 8 tests still running
        PASS [   0.174s] test-project::tests fail_expect_expanded
        PASS [   0.153s] test-project::tests fail_expect_expanded_args
        FAIL [   0.134s] test-project::tests pass

--- STDOUT:              test-project::tests pass ---

running 1 test
Running 4 macro expansion tests
test pass ... FAILED

failures:

failures:
    pass

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.08s

--- STDERR:              test-project::tests pass ---
thread 'pass' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.140s] test-project::tests pass_args

--- STDOUT:              test-project::tests pass_args ---

running 1 test
Running 1 macro expansion tests
test pass_args ... FAILED

failures:

failures:
    pass_args

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.08s

--- STDERR:              test-project::tests pass_args ---
thread 'pass_args' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.133s] test-project::tests pass_expect_expanded

--- STDOUT:              test-project::tests pass_expect_expanded ---

running 1 test
Running 4 macro expansion tests
test pass_expect_expanded ... FAILED

failures:

failures:
    pass_expect_expanded

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.08s

--- STDERR:              test-project::tests pass_expect_expanded ---
thread 'pass_expect_expanded' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.144s] test-project::tests pass_expect_expanded_args

--- STDOUT:              test-project::tests pass_expect_expanded_args ---

running 1 test
Running 1 macro expansion tests
test pass_expect_expanded_args ... FAILED

failures:

failures:
    pass_expect_expanded_args

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.09s

--- STDERR:              test-project::tests pass_expect_expanded_args ---
thread 'pass_expect_expanded_args' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        FAIL [   0.131s] test-project::tests pr61

--- STDOUT:              test-project::tests pr61 ---

running 1 test
Running 2 macro expansion tests
test pr61 ... FAILED

failures:

failures:
    pr61

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 6 filtered out; finished in 0.08s

--- STDERR:              test-project::tests pr61 ---
thread 'pr61' panicked at D:\git-repos\github\macrotest\src\expand.rs:128:9:
prepare failed: IoError(
    Os {
        code: 32,
        kind: Uncategorized,
        message: "The process cannot access the file because it is being used by another process.",
    },
)
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

        PASS [   1.853s] test-project::par_tests_regression parallel_1
------------
     Summary [   1.856s] 9 tests run: 3 passed, 6 failed, 0 skipped
        FAIL [   0.194s] test-project::par_tests_regression parallel_2
        FAIL [   0.134s] test-project::tests pass
        FAIL [   0.140s] test-project::tests pass_args
        FAIL [   0.133s] test-project::tests pass_expect_expanded
        FAIL [   0.144s] test-project::tests pass_expect_expanded_args
        FAIL [   0.131s] test-project::tests pr61
error: test run failed

[wmmc88]

It looks like its panicking here due to error deleting the target/tests/<CRATE NAME>/macrotest000X directory .

the unique names from COUNT seem to handle multiple calls to expand/expand_args within a single binary, but this won't work for multiple test binaries. I skimmed thru trybuild code and it seems like they avoid this by having a flock on the directory

[taiki-e]

This is kind of a side effect of https://github.com/eupn/macrotest/pull/69.

That said, since the target directory is shared, it should not work to actually run macrotest in parallel anyway due to the cargo's mechanism.

File lock seems to be a reasonable approach here.