search

Zac-HD / hypofuzz

Adaptive fuzzing of Hypothesis tests
https://hypofuzz.com/docs
GNU Affero General Public License v3.0
72 stars 3 forks source link

RuleBasedStateMachine tests do not work and are not documented #38

Open pspacek opened 4 months ago

pspacek commented 4 months ago

It seems that tests based on RuleBasedStateMachine are not supported. I've tried to full-text search for "RuleBased" and "stateful" in hypofuzz docs but have not seen any mention this is not supported.

After some digging I've found a mention in the sources:

# Skip state-machine classes, since they're not

but it's a bit mysterious :-)

Maybe collector could print a message when in encounters unsupported case instead of silently ignoring it?

Versions tested

Steps to reproduce

  1. Copy & paste example code from https://hypothesis.readthedocs.io/en/hypothesis-python-4.57.1/stateful.html into a file, say test_example.py.

  2. Run hypothesis fuzz test_example.py

Output

Usage: hypothesis fuzz [OPTIONS] [-- PYTEST_ARGS]
Try 'hypothesis fuzz -h' for help.

Error: No property-based tests were collected

Further details

With a slight modification to hypofuzz/interface.py we can get more verbose output. Diff:

@@ -81,8 +81,8 @@ def _get_hypothesis_tests_with_pytest(args: Iterable[str]) -> List["FuzzProcess"
             ],
             plugins=[collector],
         )
+    print(out.getvalue())  # noqa
     if ret:
-        print(out.getvalue())  # noqa
         print(f"Exiting because pytest returned exit code {ret}")  # noqa
         sys.exit(ret)
     return collector.fuzz_targets

Output:

================================ test session starts =================================
platform linux -- Python 3.11.8, pytest-8.0.2, pluggy-1.5.0
rootdir: /tmp/ste/orig
plugins: hypothesis-6.100.1, dash-2.16.1
collected 1 item

<Dir orig>
  <Module test_example.py>
    <UnitTestCase TestDBComparison>
      <TestCaseFunction runTest>
crashed in test_example.py::TestDBComparison::runTest 'function' object has no attribute 'hypothesis'

============================= 1 test collected in 0.02s ==============================
pspacek commented 4 months ago

Additional note, after commenting out print around catch-all except I get more detailed traceback:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 273, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 326, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/logging.py", line 783, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/warnings.py", line 118, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/config/__init__.py", line 1367, in pytest_collection
INTERNALERROR>     return (yield)
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 337, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/_pytest/main.py", line 810, in perform_collect
INTERNALERROR>     hook.pytest_collection_finish(session=self)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>           ^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/hypofuzz/interface.py", line 58, in pytest_collection_finish
INTERNALERROR>     fuzz = FuzzProcess.from_hypothesis_test(
INTERNALERROR>            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/tmp/ste/hypofuzz/lib/python3.11/site-packages/hypofuzz/hy.py", line 96, in from_hypothesis_test
INTERNALERROR>     given_kwargs=wrapped_test.hypothesis._given_kwargs,
INTERNALERROR>                  ^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> AttributeError: 'function' object has no attribute 'hypothesis'
Zac-HD commented 4 months ago

If someone is going to spend a few hours on this, we should probably add the logic necessary to support RuleBasedStateMachine tests instead of working around that - it's a bit fiddly but not that hard, and we'll definitely want it before HypoFuzz graduates from alpha to beta 🙂

eivindjahren commented 2 months ago

It was also confusing to me why my tests were being skipped, and after debugging the HypoFuzz _ItemCollector I realized it was because the function was silently using the monkeypatch fixture through an autouse fixture. Since the monkeypatch fixture is not autouse, but automatically applied by the autouse fixture, all tests were skipped.