search

swellaby / vscode-rust-test-adapter

Rust Test Explorer extension for VS Code
MIT License
56 stars 10 forks source link

Support output from libtest-mimic #93

Open RReverser opened 4 years ago

RReverser commented 4 years ago

I'm using libtest_mimic to autogenerate a set of tests with same supported flags as regular cargo test harness.

Unfortunately, it seems that Rust Test Explorer doesn't detect those tests in the project. I'm not sure how it performs test enumeration - does it parse source files looking for #[test]? does it run something like cargo test -- --list? - but in case it's the latter, it is supported by libtest-mimic too and should, in theory, be possible to incorporate into the Rust Test Explorer.

E.g. this is the output I'm seeing when running cargo test -- --list on my project:

     Running target\debug\deps\spec-af280007c0fc00ca.exe
tests\testsuite\address.wast:3:2: test
tests\testsuite\address.wast:223:2: test
tests\testsuite\address.wast:514:2: test
tests\testsuite\address.wast:564:2: test
tests\testsuite\align.wast:3:2: test
tests\testsuite\align.wast:4:2: test
tests\testsuite\align.wast:5:2: test
tests\testsuite\align.wast:6:2: test
tests\testsuite\align.wast:7:2: test
tests\testsuite\align.wast:8:2: test
tests\testsuite\align.wast:9:2: test
tests\testsuite\align.wast:10:2: test
tests\testsuite\align.wast:11:2: test
tests\testsuite\align.wast:12:2: test
tests\testsuite\align.wast:13:2: test
tests\testsuite\align.wast:14:2: test
tests\testsuite\align.wast:15:2: test
tests\testsuite\align.wast:16:2: test
tests\testsuite\align.wast:17:2: test
tests\testsuite\align.wast:18:2: test
tests\testsuite\align.wast:19:2: test
tests\testsuite\align.wast:20:2: test
tests\testsuite\align.wast:21:2: test
tests\testsuite\align.wast:22:2: test
tests\testsuite\align.wast:23:2: test
tests\testsuite\align.wast:24:2: test
tests\testsuite\align.wast:25:2: test
tests\testsuite\align.wast:305:2: test
tests\testsuite\align.wast:309:2: test
tests\testsuite\align.wast:313:2: test
tests\testsuite\align.wast:317:2: test
tests\testsuite\align.wast:321:2: test
tests\testsuite\align.wast:325:2: test
tests\testsuite\align.wast:329:2: test
tests\testsuite\align.wast:333:2: test
tests\testsuite\align.wast:337:2: test
tests\testsuite\align.wast:341:2: test
tests\testsuite\align.wast:345:2: test
tests\testsuite\align.wast:349:2: test
tests\testsuite\align.wast:353:2: test
tests\testsuite\align.wast:357:2: test
tests\testsuite\align.wast:362:2: test
tests\testsuite\align.wast:366:2: test
tests\testsuite\align.wast:370:2: test
tests\testsuite\align.wast:374:2: test
tests\testsuite\align.wast:378:2: test
tests\testsuite\align.wast:382:2: test
tests\testsuite\align.wast:386:2: test
tests\testsuite\align.wast:390:2: test
tests\testsuite\align.wast:394:2: test
tests\testsuite\align.wast:398:2: test
tests\testsuite\align.wast:402:2: test
tests\testsuite\align.wast:406:2: test
tests\testsuite\align.wast:410:2: test
tests\testsuite\align.wast:414:2: test
tests\testsuite\align.wast:419:2: test
tests\testsuite\align.wast:423:2: test
tests\testsuite\align.wast:427:2: test
tests\testsuite\align.wast:431:2: test
tests\testsuite\align.wast:435:2: test
tests\testsuite\align.wast:439:2: test
tests\testsuite\align.wast:443:2: test
tests\testsuite\align.wast:447:2: test
tests\testsuite\align.wast:451:2: test
tests\testsuite\align.wast:458:2: test
tests\testsuite\align.wast:854:2: test
tests\testsuite\binary-leb128.wast:2:2: test
tests\testsuite\binary-leb128.wast:7:2: test
tests\testsuite\binary-leb128.wast:12:2: test
tests\testsuite\binary-leb128.wast:18:2: test
...
calebcartwright commented 4 years ago

does it parse source files looking for #[test]? does it run something like cargo test -- --list

Yes (the latter), cargo is used for test discovery and execution. Do you have a reference project you can share that reproduces the error? Is there any output in the Test Explorer output window?

RReverser commented 4 years ago

Do you have a reference project you can share that reproduces the error? Is there any output in the Test Explorer output window?

No interesting output (aside from warnings about "example" and "bench" being unsupported types, but those are unrelated):

[2020-10-22 01:30:37.475] [INFO] Test Explorer found
[2020-10-22 01:30:37.476] [INFO] Initializing Rust adapter
[2020-10-22 01:30:37.476] [INFO] Loading Rust Tests
[2020-10-22 01:30:37.632] [WARN] Unsupported target type: example for dump
[2020-10-22 01:30:37.632] [WARN] Unsupported target type: bench for bench

The project where I'm observing this is here: https://github.com/GoogleChromeLabs/wasmbin. Just make sure to git submodule update --init after cloning, too.

calebcartwright commented 4 years ago

Thanks! The unsupported target type is interesting, will try to take a closer look tomorrow

calebcartwright commented 4 years ago

Ahh so there's no issue here, just that your project only has integration and bench tests which we don't support yet (refs #3 and https://github.com/swellaby/vscode-rust-test-adapter#current-features).

There's no tests showing in the explorer because there are no unit tests on the wasmbin nor wasmbin-derive targets

$ cargo test -p wasmbin --lib -- --list
    Finished test [unoptimized + debuginfo] target(s) in 0.06s
     Running target/debug/deps/wasmbin-63ef3693d14345ac
0 tests, 0 benchmarks
$ cargo test -p wasmbin-derive --lib -- --list
    Finished test [unoptimized + debuginfo] target(s) in 0.05s
     Running target/debug/deps/wasmbin_derive-5ddfaf408a794d84
0 tests, 0 benchmarks
$ cargo test --test spec -- --list
    Finished test [unoptimized + debuginfo] target(s) in 0.06s
     Running target/debug/deps/spec-0e584663778ea164
tests/testsuite/linking.wast:3:2: test
tests/testsuite/linking.wast:9:2: test
tests/testsuite/linking.wast:22:2: test
tests/testsuite/linking.wast:27:2: test
...
RReverser commented 4 years ago

Hmm I see. What is the practical difference between unit tests and integration tests for the Test Explorer? Do they have different output in --list or is it just that Test Explorer currently only lists tests with --lib?

If it's the latter, then it seems almost easy to just remove --lib, but maybe I'm missing something.

calebcartwright commented 4 years ago

What is the practical difference between unit tests and integration tests for the Test Explorer?

It's about correct test discovery and association, both from an organizational standpoint in the explorer but also importantly to support correctly running a given test case. There's no guarantee that there will never be any overlap between mod/test case names, so for execution we have to provide the additional args that include the target kind and in the case of a bin or test target type the associated name.

For example, if you have unit tests under a mod named tests as well as integration (or other) tests that also fall under a mod named tests. Those are separate nodes in the explorer tree that users can run individually, and we have to be able to support provide the correct args to cargo to only run what the user actually clicked on/expects

RReverser commented 4 years ago

There's no guarantee that there will never be any overlap between mod/test case names

Good point. I thought that at least for regular (not mimicked) tests the generated names include full path, and easy to distinguish between lib and integration, but appears not.

I'm guessing same applies to doing cargo test --tests since this saves us from overlap with --lib tests, but not from tests with the same name in different integration modules.

I don't suppose there is a nice way to fix this and distinguish all targets, but perhaps parsing stderr might help? I see that the combined stderr+stdlog output looks like this:

    Finished test [unoptimized + debuginfo] target(s) in 0.24s
     Running target\debug\deps\wasmbin-a65793de70add91b.exe
t: test

1 test, 0 benchmarks
     Running target\debug\deps\spec-8f9b96222b23e0f7.exe
tests\testsuite\address.wast:3:2: test
tests\testsuite\address.wast:223:2: test
tests\testsuite\address.wast:514:2: test
tests\testsuite\address.wast:564:2: test
...
     Running target\debug\deps\ttt-59f58a2ec45ad4b9.exe
t: test

1 test, 0 benchmarks
   Doc-tests wasmbin
0 tests, 0 benchmarks

By parsing the Running ... lines and extracting only the basename minus hash, you could present different targets as respective nodes, e.g. in the example above:

wasmbin
  t
spec
  tests\testsuite\address.wast:3:2
  ...
ttt
  t
calebcartwright commented 4 years ago

We already have a straightforward plan to incorporate integration tests (it's not a terribly challenging problem to solve), just haven't gotten around to it yet.

RReverser commented 4 years ago

Ah fair enough, I'll shut up with my ideas then :D

calebcartwright commented 4 years ago

Ah fair enough, I'll shut up with my ideas then :D

Oh no worries, always appreciated and apologies if I gave that impression! I was just noting that the reason we don't support this yet is because of lack of bandwidth and not due to not having a solution.

My limited open source time has been pretty tied up in other projects for a while and this one hasn't seen much love recently :smile: Hoping to get back to it in the coming weeks over the holidays.

RReverser commented 4 years ago

My limited open source time has been pretty tied up in other projects for a while and this one hasn't seen much love recently 😄

That's okay, I know the feeling. Thanks for taking a look at my issue!