Make sure the new architecture supports as much of the old api as possible.

CodyKochmann / battle_tested

Fully automated python fuzzer built to test if code actually is production ready in seconds.

MIT License

103 stars 5 forks source link

Make sure the new architecture supports as much of the old api as possible. #44

Open CodyKochmann opened 5 years ago

CodyKochmann commented 5 years ago

The following will still be supported:

[ ] the default result = fuzz(fn) basic usage
[ ] the decorator syntax
[ ] the seconds parameter
[ ] the max_tests parameter
[ ] the verbose parameter
[ ] the allow parameter

The following parameters will be remapped:

[ ] keep_testing will become exit_on_crash
[ ] strategies will become input_types

The following new parameters will be added:

[ ] input_examples - an iterable of example arguments BT will use to mutate / create test inputs
[ ] example_count - how many unique crashes / successful io BT will collect per input type combination

marcinpohl commented 5 years ago

wordsmithing:

exit_on_FIRST_crash for clarity?
strategies might be a good name to keep, for people used to Hypothesis. alias input_types maybe?
example_count does not convey ANY of the details you then explained in description. needs a much better name.

CodyKochmann commented 5 years ago

I was hesitant to change strategies but since I'm dumbing down custom input types to purely types, t seemed unfair to claim that strategies are an option.

CodyKochmann commented 5 years ago

The example_count option is tricky. I wanted to just make it a value that BT calculated on its own but with how the code was structured and how diverse the systems are that are using this tool, I wasn't too confident that BT would always make the right choice.

I'm considering abstracting away from that detail and putting something like max_memory, memory_limit or something along those lines since that variable effectively is completing the calculation of how much memory the fuzz will be able to grow to.

CodyKochmann commented 5 years ago

I am happy YOU said exit_on_first_crash because that was what was there before I committed. It just felt like a few too many characters for a variable so I wasn't sure.