search

google / oss-fuzz-gen

LLM powered fuzzing via OSS-Fuzz.
Apache License 2.0
841 stars 98 forks source link

benchmarks: add a larger set of test-to-harness targets #544

Closed DavidKorczynski closed 1 month ago

DavidKorczynski commented 1 month ago

For many projects there are more test files available, but am limiting it to 9 files for now. This should be good for assessing quality for now. There is ~160 projects in this benchmark.

DavidKorczynski commented 1 month ago

/gcbrun exp -n dk-test-to-harness-exp-2 -m vertex_ai_gemini-1-5 -i -b comparison from-test-small

DavidKorczynski commented 1 month ago

/gcbrun exp -n dk-test-to-harness-exp-3 -m vertex_ai_gemini-1-5 -i -b from-test-small

DavidKorczynski commented 1 month ago

The small is looking good https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2024-08-18-544-dk-test-to-harness-exp-3-from-test-small/index.html

Testing on the larger now

DavidKorczynski commented 1 month ago

/gcbrun exp -n dk-test-to-harness-exp-4 -m vertex_ai_gemini-1-5 -i -b from-test-large

DavidKorczynski commented 1 month ago

Large scale experiment: https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2024-08-18-544-dk-test-to-harness-exp-4-from-test-large/index.html