search

google / oss-fuzz-gen

LLM powered fuzzing via OSS-Fuzz.
Apache License 2.0
841 stars 98 forks source link

Grid-search Temperature #533

Closed DonggeLiu closed 1 month ago

DonggeLiu commented 1 month ago

08-07 experiment, with temperature 0-0.4, did not perform well either. Given we have a weekly experiment scheduled tmr and only very few temperature values were untested, I'd propose doing a "grid search" on the rest.

Current ranking is:

  1. 07-06 with constant temperature 0.4: 1042 successful builds.
  2. 08-03 with constant temperature 1: 960 builds.
  3. 08-07 with vary temperature 0-0.4: 898 builds.

The new experiment will test 0.5-0.9 so that we have some data for each.

DonggeLiu commented 1 month ago

/gcbrun exp -n dg

DonggeLiu commented 1 month ago

https://llm-exp.oss-fuzz.com/Result-reports/ofg-pr/2024-08-09-533-dg-comparison/index.html

DonggeLiu commented 1 month ago

Let me merge this for now, in case I forget. But I am happy to adjust if @mihaimaruseac has suggestions : )