search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.09k stars 266 forks source link

Fix/Remove `aspell_aspell_fuzzer_e8eb74` from tag `SBFT'23` #1992

Closed DonggeLiu closed 2 months ago

DonggeLiu commented 4 months ago

The benchmark is no longer compatible, and bug/coverage measurement failures were caused during experiments. Results: https://storage.googleapis.com/www.fuzzbench.com/reports/2024-05-16-new-bug/index.html https://storage.googleapis.com/www.fuzzbench.com/reports/2024-05-17-new-bug/index.html

aspell_aspell_fuzzer_e8eb74 is the only one causing this problem, other benchmarks are fine: https://github.com/google/fuzzbench/pull/1985#issuecomment-2120328749

ChaitanyaYeole02 commented 2 months ago

Hey @DonggeLiu, I am interest to work on this, can you please assign it to me?

DonggeLiu commented 2 months ago

Yep sure, thanks @ChaitanyaYeole02 !

ChaitanyaYeole02 commented 2 months ago

Is there any development documentation I can refer to?

DonggeLiu commented 2 months ago

Some general development documentation: https://google.github.io/fuzzbench/developing-fuzzbench/adding-a-new-benchmark/

For this case, we can try to find a newer commit with a crash found by OSS-Fuzz), verify if the crash is reproducible in FuzzBench, and update the benchmark to that commit. If none is found/reproducible in FuzzBench, we remove the benchmark (or replace it with another one). All crashes from OSS-Fuzz can be found from: https://bugs.chromium.org/p/oss-fuzz/issues/list?groupby=Proj&colspec=ID%20Type%20Component%20Status%20Reported%20Owner%20Summary&q=&can=1

ChaitanyaYeole02 commented 2 months ago

Hey @DonggeLiu , I am not able to find any bugs in OSS-Fuzz to specific aspell_aspell_fuzzer_e8eb74 benchmark.

I am trying to replicate this experiements I found over here https://github.com/google/fuzzbench/pull/1985#issuecomment-2120253997.

Am I on the right track?

DonggeLiu commented 2 months ago

Yep it appears that the experiment finished successfully sometime after this issue had been created. I wonder if it was a flaky error, I will close this issue for now. Thanks for bring this to our attention.