cucumber integration tests fail to accept standard libtest args

[sophokles73]

I am trying to add some cucumber based integration tests to an existing suite of standard rust unit tests.

The tests are run in a CI job with

cargo test --all-targets -- -Z unstable-options --report-time --format json

which fails when the first cucumber based integration test is run with:

error: unexpected argument '-Z' found

I guess this is because the cucumber test harness does not support/understand the standard libtest args. However, I haven't yet found out, how I could exclude the cucumber based integration tests from being run by the cargo test ... command.

Any hints?

[ilslv]

@sophokles73 we actually do support --format json as an argument, but it's behind a libtest cargo feature and requires Libtest Writer.

Separate book chapter describes it in more details.

[sophokles73]

Thanks for the reply. Using the libtest features seems to allow passing in -Z unstable-options --format json but not the --report-time :-(

cargo test -- -Z unstable-options --report-time --format json
...
error: unexpected argument '--report-time' found

  note: argument '--retry-after' exists

[ilslv]

@sophokles73 we currently don't support this feature, but you can ignore it with custom CLI option.

[ilslv]

Actually, I think we already report timing, just don't support CLI option. Should be a relatively easy fix, will look into it in the near future.

[sophokles73]

I use the custom cli approach to swallow the --report-time switch. Having this supported out-of-the-box would, of course, be much appreciated :-)

[ilslv]

@sophokles73 hm, it seems like --report-time option does absolutely nothing in combination with format=json, because this format always reports time.

cargo test --all-targets -- -Z unstable-options --format json
    Finished test [unoptimized + debuginfo] target(s) in 0.17s
     Running unittests src/main.rs
{ "type": "suite", "event": "started", "test_count": 0 }
{ "type": "suite", "event": "ok", "passed": 0, "failed": 0, "ignored": 0, "measured": 0, "filtered_out": 0, "exec_time": 0.00002885 }

cargo test --all-targets -- -Z unstable-options --report-time --format json
    Finished test [unoptimized + debuginfo] target(s) in 0.19s
     Running unittests src/main.rs
{ "type": "suite", "event": "started", "test_count": 0 }
{ "type": "suite", "event": "ok", "passed": 0, "failed": 0, "ignored": 0, "measured": 0, "filtered_out": 0, "exec_time": 0.000030493 }

Why do you use it? Is it some tool, that automatically passes those arguments?

[sophokles73]

I am adding cucumber tests to an existing code base that currently only uses unit tests and specifies these options for executing the tests on CI ...

[ilslv]

@sophokles73 as this crate supports only json format (there are no plans to support other formats) and format=json --report-time is equivalent to format=json, I don't think this is reasonable to add this CLI flag. It would only cause confusion and additional burden of supporting unstable feature that could be changed/broken at any time.

You should be able to remove this flag from CI. If not, working workaround with custom CLI options was found.

I'm closing this issue, but feel free to reopen in case there are question left.

[sophokles73]

I see your point. However, what you state is only true for cucumber-rs, i.e. for libtest, it actually makes a difference to add --report-time to --format=json. If you omit it then the json output for the individual tests does not contain timing information and there will only be the total execution time for all tests included.

[tyranron]

@ilslv let's handle --report-time gracefully and mention it in the Book.

[ilslv]

If you omit it then the json output for the individual tests does not contain timing information and there will only be the total execution time for all tests included.

Oh, didn't realise that. With that in mind, this feature makes sense, thanks. Implemented it in #265