search

SatelliteApplicationsCatapult / tribble

Coverage based JVM Fuzz testing tool.
Apache License 2.0
19 stars 1 forks source link

Apparent failures not saved to failed path #96

Closed mncat77 closed 6 years ago

mncat77 commented 6 years ago

I am currently using the tribble maven plugin v0.6 to test a target that can either return FuzzResult.OK, FuzzResult.FAILED, or throw an exception. Tribble frequently mentions failures in its output, e.g.

runs: 8986 fails: 1637 timeouts: 0 paths: 14 total time: [...]

However, no files are generated in the failed directory (even though it is being created at startup and files are generated in the corpus directory), which leads me to believe that I'm either misinterpreting the fails: x output, or failures are not being saved to the failed path as they should be. This occurs even with the default configuration (except for target).

It would be nice to know what exactly the fails: stat denotes and if it's possible to get the inputs used for those failed runs.

emilyselwood commented 6 years ago

Interesting. Thanks for the bug report. Do you get a single file called NoStackTrace.failed created? Normally it should only create a single output for each filename. If your FuzzTest returns FuzzResult.FAILED then there will be no stack trace (which gets hashed for failed runs to be the filename to reduce the number of overlapping outputs generated)

We should probably change this so that when there is no stack trace we use a hash of the input. Let me put together a test and do some experiments. I'll admit I haven't used the FuzzResult.FAILED output much, I usually just use OK and exceptions being thrown.

If you throw an exception rather than returning FuzzResult.FAILED do you get files written out?

emilyselwood commented 6 years ago

I think I have reproduced this. I'll have a look if I get a chance over the weekend. Else it will be Tuesday before I get to look. (we have a bank holiday here on Monday)

mncat77 commented 6 years ago

Tried throwing exceptions instead, which didn't change anything.

Returning FuzzResult.FAILED or throwing an exception unconditionally leads to the creation of files, so I'm leaning towards the fails: counter being inaccurate somehow.

mncat77 commented 6 years ago

Turns out this is working as intended and I misinterpreted FuzzResult.IGNORE.

Before running the actual test, very large inputs were filtered out (because they inherently cause timouts due to their size) using FuzzResult.IGNORE, which I now know to count as a failure. Replacing IGNORE with OK gives the expected output of 0 fails.

emilyselwood commented 6 years ago

I'm going to reopen this. I think I want to change that behaviour so that ignore is not counted in the failed stats. As you have found this is confusing it does not meet the easy to use design goal. I'll also see about adding more documentation around the FuzzResult to make things clearer.

Thank you for the update. This now makes a lot more sense.

emilyselwood commented 6 years ago

I've got a few bits to finish off before I can push the 0.7 release. But the next release will now count ignores separately in the stats.