search

google / oss-fuzz-gen

LLM powered fuzzing via OSS-Fuzz.
Apache License 2.0
780 stars 92 forks source link

benchmarks: update JVM benchmarks #445

Closed DavidKorczynski closed 1 week ago

DavidKorczynski commented 1 week ago

Changes the JVM benchmarks to be split into three buckets: 1) jvm-all: all java projects in oss-fuzz 2) jvm-medium: a smaller set of ~20 projects, which is useful for testing semantic changes in prompts without needing to check all projects. 3) jvm-small: set of three projects that can be used to test changes in infrastructure to ensure no infra regressions.

DavidKorczynski commented 1 week ago

Generated using [far-reach-low-coverage,jvm-public-candidates,easy-params-far-reach] and max 6 targets per oracle

DavidKorczynski commented 1 week ago

/gcbrun exp -n dk-test-jvm3234 -m vertex_ai_gemini-1-5 -b jvm-small

DavidKorczynski commented 1 week ago

Experiment looks good

DavidKorczynski commented 1 week ago

/gcbrun exp -n dk-test-infra5205 -m vertex_ai_gemini-1-5 -b jvm-small -i

DavidKorczynski commented 1 week ago

Small experiment is looking great https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2024-07-11-445-dk-test-infra5205-jvm-small/index.html, let's do a full run

DavidKorczynski commented 1 week ago

/gcbrun exp -n dk-test-infra5209 -m vertex_ai_gemini-1-5 -b jvm-all -i