google / oss-fuzz-gen

LLM powered fuzzing via OSS-Fuzz.
Apache License 2.0
780 stars 92 forks source link

experimental: add post-processing logic #440

Closed DavidKorczynski closed 1 week ago

DavidKorczynski commented 1 week ago

This is needed to easily digest the results from a larger scale run

DavidKorczynski commented 1 week ago

Sample run:

$ cat ./targets.txt 
https://github.com/gregjesl/simpleson
https://github.com/memononen/nanosvg
https://github.com/codeplea/tinyexpr
$ python3 ./runner.py -o ../../../oss-fuzz-3/ -i ./targets.txt -t 6 -m openai
....
...
$ python3 ./post-process.py ../../../oss-fuzz-3                                                        
[post-process.py:163]: Analysing: ../../../oss-fuzz-3/build/out/temp-project-0
[post-process.py:174]: Results: https://github.com/gregjesl/simpleson
[post-process.py:175]: - Language: c++
[post-process.py:176]: - Functions from Fuzz Introspector: 108
[post-process.py:178]: - Harnesses generated: 7
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-0/autogen-results/test-fuzz-build-2-fuzzgen-1/fuzz-run.err :: 18 :: 221 :: [asan errors: 1] :: coverage: 28.930818 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-0/autogen-results/test-fuzz-build-1-fuzzgen-8/fuzz-run.err :: 14 :: 216 :: [asan errors: 1] :: coverage: 31.238779 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-0/autogen-results/test-fuzz-build-1-fuzzgen-1/fuzz-run.err :: 18 :: 196 :: [asan errors: 1] :: coverage: 26.145553 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-0/autogen-results/test-fuzz-build-0-fuzzgen-1/fuzz-run.err :: 18 :: 160 :: [asan errors: 1] :: coverage: 23.899371 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-0/autogen-results/test-fuzz-build-0-fuzzgen-8/fuzz-run.err :: 14 :: 159 :: [asan errors: 1] :: coverage: 24.955117 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-0/autogen-results/test-fuzz-build-2-fuzzgen-8/fuzz-run.err :: 14 :: 105 :: [asan errors: 1] :: coverage: 29.174147 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-0/autogen-results/test-fuzz-build-2-fuzzgen-0/fuzz-run.err :: 2 :: 10 :: [asan errors: 0] :: coverage: 2.066487 
[post-process.py:163]: Analysing: ../../../oss-fuzz-3/build/out/temp-project-1
[post-process.py:174]: Results: https://github.com/memononen/nanosvg
[post-process.py:175]: - Language: c
[post-process.py:176]: - Functions from Fuzz Introspector: 173
[post-process.py:178]: - Harnesses generated: 16
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-1-fuzzgen-1/fuzz-run.err :: 25 :: 862 :: [asan errors: 0] :: coverage: 55.141727 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-1-fuzzgen-7/fuzz-run.err :: 2 :: 818 :: [asan errors: 0] :: coverage: 39.282186 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-0-fuzzgen-7/fuzz-run.err :: 2 :: 816 :: [asan errors: 0] :: coverage: 47.836326 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-2-fuzzgen-1/fuzz-run.err :: 25 :: 746 :: [asan errors: 0] :: coverage: 38.991430 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-2-fuzzgen-13/fuzz-run.err :: 32 :: 735 :: [asan errors: 0] :: coverage: 55.511551 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-3-fuzzgen-7/fuzz-run.err :: 2 :: 726 :: [asan errors: 0] :: coverage: 50.576227 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-0-fuzzgen-1/fuzz-run.err :: 25 :: 717 :: [asan errors: 0] :: coverage: 36.564526 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-0-fuzzgen-13/fuzz-run.err :: 32 :: 703 :: [asan errors: 0] :: coverage: 38.289086 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-1-fuzzgen-13/fuzz-run.err :: 32 :: 657 :: [asan errors: 0] :: coverage: 44.092409 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-2-fuzzgen-7/fuzz-run.err :: 2 :: 648 :: [asan errors: 0] :: coverage: 41.389529 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-3-fuzzgen-13/fuzz-run.err :: 32 :: 628 :: [asan errors: 0] :: coverage: 39.834983 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-3-fuzzgen-1/fuzz-run.err :: 25 :: 624 :: [asan errors: 0] :: coverage: 41.957811 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-0-fuzzgen-0/fuzz-run.err :: 2 :: 358 :: [asan errors: 0] :: coverage: 26.456739 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-3-fuzzgen-0/fuzz-run.err :: 2 :: 328 :: [asan errors: 0] :: coverage: 37.590520 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-2-fuzzgen-0/fuzz-run.err :: 2 :: 302 :: [asan errors: 0] :: coverage: 27.649770 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-1/autogen-results/test-fuzz-build-1-fuzzgen-0/fuzz-run.err :: 2 :: 284 :: [asan errors: 0] :: coverage: 31.566820 
[post-process.py:163]: Analysing: ../../../oss-fuzz-3/build/out/temp-project-2
[post-process.py:174]: Results: https://github.com/codeplea/tinyexpr
[post-process.py:175]: - Language: c
[post-process.py:176]: - Functions from Fuzz Introspector: 65
[post-process.py:178]: - Harnesses generated: 14
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-2-fuzzgen-15/fuzz-run.err :: 30 :: 180 :: [asan errors: 0] :: coverage: 34.369115 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-1-fuzzgen-15/fuzz-run.err :: 30 :: 170 :: [asan errors: 0] :: coverage: 33.081285 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-2-fuzzgen-16/fuzz-run.err :: 28 :: 169 :: [asan errors: 0] :: coverage: 33.084809 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-1-fuzzgen-21/fuzz-run.err :: 30 :: 168 :: [asan errors: 0] :: coverage: 34.463277 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-1-fuzzgen-16/fuzz-run.err :: 28 :: 162 :: [asan errors: 0] :: coverage: 33.208606 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-1-fuzzgen-4/fuzz-run.err :: 28 :: 161 :: [asan errors: 0] :: coverage: 32.645403 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-2-fuzzgen-4/fuzz-run.err :: 28 :: 161 :: [asan errors: 0] :: coverage: 32.710280 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-2-fuzzgen-21/fuzz-run.err :: 30 :: 159 :: [asan errors: 0] :: coverage: 34.146341 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-1-fuzzgen-3/fuzz-run.err :: 30 :: 152 :: [asan errors: 0] :: coverage: 34.277620 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-2-fuzzgen-9/fuzz-run.err :: 30 :: 152 :: [asan errors: 0] :: coverage: 32.956685 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-2-fuzzgen-3/fuzz-run.err :: 30 :: 151 :: [asan errors: 0] :: coverage: 33.019755 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-1-fuzzgen-9/fuzz-run.err :: 30 :: 151 :: [asan errors: 0] :: coverage: 34.310019 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-1-fuzzgen-22/fuzz-run.err :: 28 :: 50 :: [asan errors: 0] :: coverage: 32.833021 
[post-process.py:180]: ../../../oss-fuzz-3/build/out/temp-project-2/autogen-results/test-fuzz-build-2-fuzzgen-22/fuzz-run.err :: 28 :: 43 :: [asan errors: 0] :: coverage: 32.710280 
DavidKorczynski commented 1 week ago

/gcbrun skip

DavidKorczynski commented 1 week ago

/gcbrun skip