google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.11k stars 269 forks source link

edges_found and edges_covered mismatch #1955

Open harrison4ride opened 9 months ago

harrison4ride commented 9 months ago

Hi all,

I am using aflplusplus and a custom afl++ fuzzer to do the experiment. During fuzzbench experiment, afl++ will generate plot_data record: # relative_time, cycles_done, cur_item, corpus_count, pending_total, pending_favs, map_size, saved_crashes, saved_hangs, max_depth, execs_per_sec, total_execs, edges_found Fuzzbench will use snapshot to evaluate the fuzzer performance recording: ,git_hash,experiment_filestore,experiment,fuzzer,benchmark,time_started,time_ended,trial_id,time,edges_covered,fuzzer_stats,crash_key,bugs_covered

And I found in the same experiment, edges_found is less than edges_covered. Any idea why? Additionally, the edges_found in custom_aflpp is higher than edges_found in aflpp, but edges_covered in custom_aflpp is less than edges_found. Here is the shortcut of raw data: plot_data of custom_aflpp:

Relative Time Cycles Done Current Item Corpus Count Pending Total Pending Favs Map Size Saved Crashes Saved Hangs Max Depth Executions Per Second Total Executions Edges Found
50398 23 128 1699 232 0 27.35% 0 0 13 4849.04 304726213 1588
50403 23 107 1699 232 0 27.35% 0 0 13 5302.55 304753210 1588
50408 23 1036 1699 232 0 27.35% 0 0 13 5249.57 304779127 1588
50413 23 1093 1699 232 0 27.35% 0 0 13 5162.56 304807198 1588
50418 23 107 1699 232 0 27.35% 0 0 13 4887.70 304831321 1588

plot_data of aflpp_baseline:

Relative Time Cycles Done Current Item Corpus Count Pending Total Pending Favs Map Size Saved Crashes Saved Hangs Max Depth Executions Per Second Total Executions Edges Found
56686 611 1673 1913 16 0 26.73% 0 0 15 16005.00 896729564 1552
56691 611 821 1913 16 0 26.73% 0 0 15 15045.84 896808391 1552
56696 611 366 1913 16 0 26.73% 0 0 15 16217.05 896887749 1552
56701 611 1473 1913 16 0 26.73% 0 0 15 16599.39 896977824 1552
Edge covered of custom_aflpp: Relative time Trial_id edge_covered
45900 28 1978
46800 28 1978
47700 28 1978
48600 28 1978
49500 28 1978
Edge covered of aflpp_baseline: Relative time Trial_id edge_covered
54900 13 2000
55800 13 2000
56700 13 2000
57600 13 2000