search

CodyKochmann / battle_tested

Fully automated python fuzzer built to test if code actually is production ready in seconds.
MIT License
103 stars 5 forks source link

fuzz(int) fails on "generator raised StopIteration" #41

Open mlao-pdx opened 5 years ago

mlao-pdx commented 5 years ago

Hi,

Not sure what is causing the issue, but trying to run the below on a Mac with Python 3.7 virtual env with Hydrogen/Atom, and battle-tested ==2017.11.18.1, I run into issues. The test count (here 1752) creeps up with each run with slightly different increments, probably due to the test case shrinking and caching that happens underneath the sheets with Hypothesis. When setting the count to or below the achieved test count, say 1700), it runs without error. Slightly above the previously achieved test count, say 1760, I might get lucky. Far above, like the 10K, then it will fail.

Arnoud.

Pipfile:

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]
ipykernel = "*"
pytest = "*"
hypothesis = "*"
battle-tested = "*"

[packages]
behave = "*"
python-docx = "*"
pandas = "*"
mistletoe = "*"

[requires]
python_version = "3.7"

Code run and its redacted result:

from battle_tested import fuzz
fuzz(int, max_tests=10000)

testing: int()
tests: 1752      speed:   3244/sec  avg:  3148/sec in 6s    
compiling results...
Traceback (most recent call last):

  File "<ipython-input-70-ae871cfb590f>", line 1, in <module>
    fuzz(int, max_tests=10000)

  File "/Users/REDACTED/.local/share/virtualenvs/REDACTED-REDACTED/lib/python3.7/site-packages/battle_tested/__init__.py", line 1584, in fuzz
    for test_args in test_gen:
  File "/Users/REDACTED/.local/share/virtualenvs/REDACTED-REDACTED/lib/python3.7/site-packages/battle_tested/__init__.py", line 1313, in generate_examples
    out = [next(garbage) for i in range(args_needed)]
  File "/Users/REDACTED/.local/share/virtualenvs/REDACTED-REDACTED/lib/python3.7/site-packages/battle_tested/__init__.py", line 1313, in <listcomp>
    out = [next(garbage) for i in range(args_needed)]
  File "/Users/REDACTED/.local/share/virtualenvs/REDACTED-REDACTED/lib/python3.7/site-packages/battle_tested/__init__.py", line 413, in multiprocess_garbage
    yield fast_alternative_next()
  File "/Users/REDACTED/.local/share/virtualenvs/REDACTED-REDACTED/lib/python3.7/site-packages/battle_tested/__init__.py", line 314, in garbage
    yield next(strat)
  File "/Users/REDACTED/.local/share/virtualenvs/REDACTED-REDACTED/lib/python3.7/site-packages/battle_tested/__init__.py", line 287, in dicts
    yield { k:v for k,v in gen.chunks(islice(strat,length*2), 2) }
  File "/Users/REDACTED/.local/share/virtualenvs/REDACTED-REDACTED/lib/python3.7/site-packages/battle_tested/__init__.py", line 287, in <dictcomp>
    yield { k:v for k,v in gen.chunks(islice(strat,length*2), 2) }
  File "/Users/REDACTED/.local/share/virtualenvs/REDACTED-REDACTED/lib/python3.7/site-packages/generators/chunks.py", line 39, in chunks
    for chunk in pipeline:

RuntimeError: generator raised StopIteration
CodyKochmann commented 5 years ago

This is unfortunate.. But interesting! :) Nice find.

This specific piece of the fuzzing engine is being overhauled in the next version of BT so things should straighten themselves out soon.

Seeing this helps a lot though and really helps nail in a couple decisions for the next version.

Thanks for investigating!

mlao-pdx commented 5 years ago

You are welcome. Hope it helps to make a better product for all of us.

Thank you for creating battle-tested.