pschanely / hypothesis-crosshair

Level-up your Hypothesis tests with CrossHair
MIT License
8 stars 0 forks source link

`NotDeterministic` with `st.text()` with no decisions made #6

Closed tybug closed 4 months ago

tybug commented 4 months ago

I think this only happens under certain conditions since https://github.com/pschanely/CrossHair/issues/263 uses st.text() and works fine. Maybe when no meaningful decisions are made on the symbolic?

This issue also occurs with st.characters(). Found while trying to run hypothesis' test suite under crosshair (https://github.com/HypothesisWorks/hypothesis/pull/4022#pullrequestreview-2146044505). I'm tentatively calling this a crosshair-side issue, but if you discover otherwise let me know 😄

@settings(backend="crosshair", deadline=None)
@given(st.text())
def f(s):
    pass
f()

My debug output:

1272686.164|               |pre_path_hook() No coverage biasing in effect. ( 0  code locations)
1272686.165|             |condition_parser() Using parsers:  []
1272686.166|            |per_test_case_context_manager() starting iteration 1
1272686.168|                    |_next_name() Drawing str_01
1272686.172|                 |find_model_value()  *** Begin Not Deterministic Debug *** 
1272686.173|                 |find_model_value() Model value node expected; found <class 'crosshair.statespace.DeatchedPathNode'> instead.
1272686.173|                 |find_model_value()   Traceback:  (<module> sandbox4.py:36) (f sandbox4.py:33) (wrapped_test core.py:1623) (run_engine core.py:1151) (run engine.py:736) (_run engine.py:1198) (generate_new_examples engine.py:881) (cached_test_function engine.py:1432) (test_function engine.py:434) (__stoppable_test_function engine.py:307) (_execute_once_for_engine core.py:1047) (execute_once core.py:961) (__exit__ contextlib.py:144) (per_test_case_context_manager crosshair_provider.py:110) (deep_realize core.py:251) (deepcopyext copyext.py:37) (__ch_realize__ builtinslib.py:3106) (__index__ builtinslib.py:1113) (find_model_value statespace.py:937) (test_stack util.py:221)
1272686.173|                 |find_model_value()  *** End Not Deterministic Debug *** 
1272686.174|            |per_test_case_context_manager() ended iteration (exception: NotDeterministic: ) (per_test_case_context_manager crosshair_provider.py:110) (deep_realize core.py:251) (deepcopyext copyext.py:37) (__ch_realize__ builtinslib.py:3106) (__index__ builtinslib.py:1113) (find_model_value statespace.py:939)
1272686.174|            |per_test_case_context_manager() no decisions made; ignoring this iteration
1272686.197|         |export_value() WARNING: export_value() requested before test case complered (<module> sandbox4.py:36) (f sandbox4.py:33) (wrapped_test core.py:1623) (run_engine core.py:1151) (run engine.py:736) (_run engine.py:1198) (generate_new_examples engine.py:881) (cached_test_function engine.py:1432) (test_function engine.py:458) (post_test_case_hook crosshair_provider.py:286) (export_value crosshair_provider.py:277) (test_stack util.py:221)
1272686.203|                       |__init__() CrosshairInternal 
1272686.204|                       |__init__()  Stack trace:
  File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 36, in <module>
    f()
  File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 33, in f
    @given(st.text())
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1623, in wrapped_test
    state.run_engine()
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1151, in run_engine
    runner.run()
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 736, in run
    self._run()
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 1198, in _run
    self.generate_new_examples()
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 881, in generate_new_examples
    zero_data = self.cached_test_function(bytes(BUFFER_SIZE))
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 1432, in cached_test_function
    self.test_function(data)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 510, in test_function
    self.__stoppable_test_function(data)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/engine.py", line 307, in __stoppable_test_function
    self._test_function(data)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1041, in _execute_once_for_engine
    result = self.execute_once(data)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 964, in execute_once
    result = self.test_runner(data, run)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 737, in default_executor
    return function(data)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 875, in run
    kw, argslices = context.prep_args_kwargs_from_strategies(
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/control.py", line 157, in prep_args_kwargs_from_strategies
    obj = check(self.data.draw(s, observe_as=f"generate:{k}"))
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2423, in draw
    return strategy.do_draw(self)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/lazy.py", line 167, in do_draw
    return data.draw(self.wrapped_strategy)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2417, in draw
    return strategy.do_draw(self)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/strategies/_internal/strings.py", line 117, in do_draw
    return data.draw_string(
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2182, in draw_string
    node = self._pop_ir_tree_node("string", kwargs, forced=forced)
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 2330, in _pop_ir_tree_node
    if not ir_value_permitted(node.value, node.ir_type, kwargs):
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/internal/conjecture/data.py", line 1089, in ir_value_permitted
    if len(value) < kwargs["min_size"]:
  File "/opt/homebrew/lib/python3.12/site-packages/crosshair/libimpl/builtinslib.py", line 1112, in __index__
    space = context_statespace()
  File "/opt/homebrew/lib/python3.12/site-packages/crosshair/statespace.py", line 247, in context_statespace
    raise CrosshairInternal
  File "/opt/homebrew/lib/python3.12/site-packages/crosshair/util.py", line 594, in __init__
    debug(" Stack trace:\n" + "".join(traceback.format_stack()))

Traceback (most recent call last):
  File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 36, in <module>
    f()
  File "/Users/tybug/Desktop/Liam/coding/sandbox/hypothesis_/sandbox4.py", line 33, in f
    @given(st.text())
                   ^^^
  File "/Users/tybug/Desktop/Liam/coding/hypothesis/hypothesis-python/src/hypothesis/core.py", line 1656, in wrapped_test
    raise the_error_hypothesis_found
  File "/opt/homebrew/lib/python3.12/site-packages/crosshair/libimpl/builtinslib.py", line 1112, in __index__
    space = context_statespace()
            ^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/lib/python3.12/site-packages/crosshair/statespace.py", line 247, in context_statespace
    raise CrosshairInternal
crosshair.util.CrosshairInternal
pschanely commented 4 months ago

Ah yes, I've got some hacks on the plugin side to avoid these, but it must not be quite right. I'll investigate tomorrow!

pschanely commented 4 months ago

Yup, a dumb bug; fixed and added a regression test. Can you try hypothesis-crosshair 0.0.5 and confirm that helps on your side too?

tybug commented 4 months ago

Looks good here 👍