PropCheck.Test.TargetTreeTest can time out

evnu commented 3 years ago

PropCheck.Test.TargetTreeTest sometimes times out on my machine when running repeatedly with the same settings as CI:

while PROPCHECK_NUMTESTS=200 PROPCHECK_SEARCH_STEPS=1000 mix test test/targeted_tree_test.exs:43; do echo ok; done

Error:

  1) property A left-heavy tree (PropCheck.Test.TargetTreeTest)
     test/targeted_tree_test.exs:43
     ** (ExUnit.TimeoutError) property timed out after 60000ms. You can change the timeout:

       1. per test by setting "@tag timeout: x" (accepts :infinity)
       2. per test module by setting "@moduletag timeout: x" (accepts :infinity)
       3. globally via "ExUnit.start(timeout: x)" configuration
       4. by running "mix test --timeout x" which sets timeout
       5. or by running "mix test --trace" which sets timeout to infinity
          (useful when using IEx.pry/0)

     where "x" is the timeout given as integer in milliseconds (defaults to 60_000).

     code: property "A left-heavy tree" do
     stacktrace:
       (stdlib 3.14.1) gen.erl:208: :gen.do_call/4
       (stdlib 3.14.1) gen_server.erl:234: :gen_server.call/2
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper_gen.erl:199: :proper_gen.generate/3
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper_gen.erl:137: :proper_gen.generate/1
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper_gen.erl:201: :proper_gen.generate/3
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper_gen.erl:137: :proper_gen.generate/1
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper_gen.erl:201: :proper_gen.generate/3
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper_gen.erl:137: :proper_gen.generate/1
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper_gen.erl:124: :proper_gen.safe_generate/1
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper.erl:1349: :proper.perform_search/8
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper.erl:1443: :proper.run/3
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper.erl:1303: :proper.perform/7
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper.erl:1179: :proper.inner_test/2
       (proper 1.4.0) /home/evnu/tools/propcheck/deps/proper/src/proper.erl:1170: :proper.test/2
       (propcheck 1.3.1-dev) lib/properties.ex:170: PropCheck.Properties.execute_property/4
       test/targeted_tree_test.exs:43: (test)
       (ex_unit 1.11.3) lib/ex_unit/runner.ex:391: ExUnit.Runner.exec_test/1
       (stdlib 3.14.1) timer.erl:166: :timer.tc/1
       (ex_unit 1.11.3) lib/ex_unit/runner.ex:342: anonymous fn/4 in ExUnit.Runner.spawn_test_monitor/4

Judging from the . that are written when using PROPCHECK_VERBOSE, test iterations slow down after a few tries. It seems that this depends on the seed used for running the tests. After finding the instance above, I was able to reproduce immediately again by using the seed from the test:

PROPCHECK_VERBOSE=1 PROPCHECK_NUMTESTS=200 PROPCHECK_SEARCH_STEPS=1000 mix test test/targeted_tree_test.exs:43 --seed 359411

evnu commented 3 years ago

I think the input list to generate the tree might just be too large. I added IO.inspect(length(l)) to the tree generator:

..snip..
.[length: 320076]
.[length: 354481]
.[length: 331755]
.[length: 437763]
.[length: 338493]
.[length: 381151]
.[length: 418301]
.[length: 281827]

@alfert to avoid timeouts, we could keep the list smaller, or pre-sort it (but that kind-of defeats the purpose of the insertions into the binary tree). Another option would be to make the examples more amenable to the insertion by splitting into chunks and sorting within the chunks.

alfert commented 3 years ago

Simply reduced the amount of search steps (1.000 -> 600). Seems to help

alfert / propcheck

PropCheck.Test.TargetTreeTest can time out #196