Zac-HD / hypofuzz

Adaptive fuzzing of Hypothesis tests
https://hypofuzz.com/docs
GNU Affero General Public License v3.0
76 stars 3 forks source link

Another issue with shrinking #23

Closed tmr232 closed 1 year ago

tmr232 commented 1 year ago

There I go again... But this time, it did find a real crashing input for my program!

As for the issue -


Running with 23.5.2.

When running the following test:

@given(text())
def test_trivial(s):
    raise Exception()

I get the following error:

Traceback (most recent call last):
  File "C:\Users\tamir\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Users\tamir\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Code\crafting-interpreters-py\.venv\Lib\site-packages\hypofuzz\interface.py", line 91, in _fuzz_several
    fuzz_several(*tests)
  File "C:\Code\crafting-interpreters-py\.venv\Lib\site-packages\hypofuzz\hy.py", line 390, in fuzz_several
    t.run_one()
  File "C:\Code\crafting-interpreters-py\.venv\Lib\site-packages\hypofuzz\hy.py", line 214, in run_one
    shrinker.shrink()
  File "C:\Code\crafting-interpreters-py\.venv\Lib\site-packages\hypothesis\internal\conjecture\shrinker.py", line 442, in shrink
    self.explain()
  File "C:\Code\crafting-interpreters-py\.venv\Lib\site-packages\hypothesis\internal\conjecture\shrinker.py", line 511, in explain
    seen_passing_buffers = self.engine.passing_buffers(
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'EngineStub' object has no attribute 'passing_buffers'
Zac-HD commented 1 year ago

😅

Thanks for the quick reports! HypoFuzz is turning out to be ironically under-tested given recent changes to the Hypothesis internals, but I'm at least adding regression tests as I go... and will fix this one after dinner.

Zac-HD commented 1 year ago

OK, give 23.05.3 a go - I think I've fixed it this time; if not please let me know again!

At any rate it's working better than it was before and has some more tests (again), so we're on the right track...

tmr232 commented 1 year ago

Looks good so far :)

I assume the following exception is the desired outcome:

Traceback (most recent call last):
  File "C:\Users\tamir\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Users\tamir\AppData\Local\Programs\Python\Python311\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Code\crafting-interpreters-py\.venv\Lib\site-packages\hypofuzz\interface.py", line 91, in _fuzz_several
    fuzz_several(*tests)
  File "C:\Code\crafting-interpreters-py\.venv\Lib\site-packages\hypofuzz\hy.py", line 418, in fuzz_several
    raise Exception("Found failures for all tests!")
Exception: Found failures for all tests!
Zac-HD commented 1 year ago

Yep, that's the expected behaviour, although I'd like it to be smarter eventually ala #3 and #12.