search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.11k stars 269 forks source link

Add method to check if benchmarks are mixed #1977

Closed gustavogaldinoo closed 6 months ago

gustavogaldinoo commented 6 months ago

Currently, Fuzzbench fails to generate a report if benchmarks are a mix between bug and coverage benchmarks.

This PR adds a method to check if benchmarks are a mix between bug and coverage benchmarks before starting the experiment, to spot the problem early and avoid wasting time.

DonggeLiu commented 6 months ago

Why isn't the existing check sufficient?

I recall the current code only checks benchmark-type consistency at the end of experiments and cannot generate a report if inconsistent.

This warns users earlier and helps save time and computation power.

jonathanmetzman commented 6 months ago

Why isn't the existing check sufficient?

I recall the current code only checks benchmark-type consistency at the end of experiments and cannot generate a report if inconsistent.

This warns users earlier and helps save time and computation power.

For sure checking ahead of the experiment is important. But if we aren't doing this currently, there's a bug to track down here that this PR hasn't fixed right?

jonathanmetzman commented 6 months ago

Why isn't the existing check sufficient?

I recall the current code only checks benchmark-type consistency at the end of experiments and cannot generate a report if inconsistent. This warns users earlier and helps save time and computation power.

For sure checking ahead of the experiment is important. But if we aren't doing this currently, there's a bug to track down here that this PR hasn't fixed right?

Ah actually I see where this might be fixed. Gustavo do you know why this didn't work previously?