search

proptest-rs / proptest

Hypothesis-like property testing for Rust
Apache License 2.0
1.63k stars 152 forks source link

Nested `proptest!{}`'s cause run time to rise exponentially #437

Open target-san opened 3 months ago

target-san commented 3 months ago

Due to how proptest!{ |(... stats ...)| { ...} } is implemented, it spawns new nested runner which in turn runs yet another num_cases. Thus, with default number of cases being 256, one nested proptest would do 65536 runs, and double nested one would do 16777216 (!!!) runs.

While I clearly understand why this happens, it was quite a surprise to me. I needed test which would perform nontrivial multistep initialization of certain engine, so I got 2 nested macros. And my test was running for like 246 seconds. When I replaced second nesting with smart-ish deduction of parameter I needed to generate, run time dropped from 246 to 3.7 seconds.

I see two problems here:

  1. Execution cost of nesting proptest! {} is IMO not articulated at all. Or I miss that warning completely.
  2. Doing whole multistage generation in parameters section, with all its composition, is quite cumbersome. Or, again, I missed that part completely.

Thoughts?

EDIT:
Maybe consider splitting proptest! macro into three? One creates outer runner and cannot be nested, another creates nested runner but does only single run (no exponent), third one behaves like old nested case?

matthew-russo commented 3 months ago

Thanks for reaching out, can you share an example of your tests? I don't think I have a good sense of what you're attempting to do

target-san commented 3 months ago

Sorry for a bit messy initial description.

In general, my scenario is multi-stage data generation since ranges for some values depend on the state of object being initialized. Some observations so far:

  1. Chaining prop_flat_map is quite cumbersome and can easily bring debug output of object being initialized into scope. So I use nested proptest! {} blocks.
  2. Each proptest! {} block spawns separate TestRunner with default settings. Which is fine overall yet introduces nested run of multiple cases, 256 by default. As a result, with 1 nested block we get 65536 runs, with 2 - 16777216 (!) runs. This may be a bit unexpected. Maybe some kind of big WARNING in macro documentation is needed. I thought initially about additional macro which performs exactly 1 nested run but that may be unnecessary.
  3. Issue with cases number multiplication may be mitigated with Config::with_cases, although it's not very reliable. Configuration gets its values overridden by environment variables after it was created by user. Someone runs all this with PROPTEST_CASES=200 and you get back to almost square one. Maybe it's worth making Config read default values from env when it's created?