search

google / fuzzbench

FuzzBench - Fuzzer benchmarking as a service.
https://google.github.io/fuzzbench/
Apache License 2.0
1.11k stars 270 forks source link

libjpeg-turbo getting less coverage? #2010

Open DanBlackwell opened 3 months ago

DanBlackwell commented 3 months ago

Not sure if this is an issue on my end, but I've noticed that libjpeg-turbo is doing significantly worse than it was previously. I've spent a bunch of time today trying to figure out what I broke, but then I spotted that it also seems to affect AFL++ which may suggest it's not on my end.

Looking at here: 2024-07-23-aflpp, aflplusplus is stuck on libjpeg-turbo. where as here: 2024-06-07-bases-aflpp, it seems fine.

With my stuff, I saw that this is fine: 2024-06-19-prescientfuzz. But this is stuck: 2024-07-03-prescientfuzz.

Looking at the logs, it almost seems like the starting corpus has changed?

From 2024-06-19:

[Testcase #0] run time: 0h-0m-8s, clients: 1, corpus: 476, objectives: 0, executions: 3047, exec/sec: 538.2, edges: 2501/28005 (8%)
We imported 476 inputs from disk.

From 2024-07-03:

[Testcase #0] run time: 0h-0m-13s, clients: 1, corpus: 425, objectives: 0, executions: 3050, exec/sec: 324.2, edges: 1828/16258 (11%)
We imported 425 inputs from disk.

EDIT: the two above have different numbers of total edges; was the fuzzing harness changed or something?

I do wonder if it's related to this oss_fuzz_corpus flag (https://github.com/google/fuzzbench/pull/1999) somehow. Maybe this is already fixed and I just need to update the base for my PR. Perhaps just running libafl and aflplusplus without using docker's cache (even for just an hour) would be enough to do a quick sanity check. What seems odd though, is that this repo had no commits on main between june 19 and july 3...

DanBlackwell commented 3 months ago

Also, the new experiment from https://github.com/google/fuzzbench/pull/1985#issuecomment-2107526658, report: here has the low coverage numbers too (for all fuzzers)