search

camshaft / bolero

property testing and verification front-end for Rust
https://camshaft.github.io/bolero
MIT License
183 stars 15 forks source link

`cargo test` does not always replay corpus #243

Open vasumv opened 2 weeks ago

vasumv commented 2 weeks ago

Hello!

I wanted to replay corpus using cargo test (documented here) but was noticing that it will not fully replay a corpus, and always terminates after 1 second.

In the case that the corpus is small, the corpus will be fully replayed and then random inputs will be generated for the remainder (you can verify this with any of the tests in examples or the fibonacci tutorial). In the case that the corpus is large, only a subset of the inputs will be run until the test terminates after 1 second.

Is there a specific flag or profile I need to ensure that the corpus and only the corpus is replayed when running cargo test? Sorry if I'm missing something simple here.

camshaft commented 2 weeks ago

Yeah this is due to the change made in https://github.com/camshaft/bolero/pull/206/files#diff-e3f9b342c59a9db62c2f297cc5bfd9823e8f90b2833b234edce7c46b865dc333R31, where it currently defaults to running tests for at most a second. If you want to increase the default you can specify a time in the harness using .with_test_time. Additionally, specifying the number of iterations also disables the time limit.

That being said, we should probably document this better.

camshaft commented 2 weeks ago

Additionally, it might be better to always replay the corpus have the time limit only apply to the random iterations. Not sure.

vasumv commented 2 weeks ago

I see, thanks for the quick response!

So if I wanted to just do this in cmdline without changing the test harness, I'd get the exact size of the corpus, and use BOLERO_RANDOM_ITERATIONS=<size_of_corpus> cargo test (which seems to have worked).

camshaft commented 2 weeks ago

Yeah the environment variable should work! One thing to note, though, is the BOLERO_RANDOM_ITERATIONS value is on top of the corpus replay. So if you have 5 corpus files and then specify BOLERO_RANDOM_ITERATIONS=6, you'll end up with 11 runs.

vasumv commented 2 weeks ago

Oh gotcha, thanks for the clarification. So really what I want is BOLERO_RANDOM_ITERATIONS=0.