search

google / oss-fuzz

OSS-Fuzz - continuous fuzzing for open source software.
https://google.github.io/oss-fuzz
Apache License 2.0
10.32k stars 2.2k forks source link

check_build passes although fuzzer is broken #5838

Open rlohning opened 3 years ago

rlohning commented 3 years ago

Hi,

I was surprised that six open issues in project Qt (for reference see below) were closed on May 21st, while I wasn't aware of any fix there. I could find out that afl_qt_qtsvg_svg_qsvgrenderer_render and libFuzzer_qt_qtsvg_svg_qsvgrenderer_render stopped running on that day as I can see in my fuzzer statistics.

While I assume that this was caused by a change in Qt's code, I wondered: Shouldn't oss-fuzz have reported this problem instead of closing the issues?

In the docker image, I can build the fuzzer successfully using:

python infra/helper.py build_fuzzers --sanitizer address --engine libfuzzer --architecture i386 qt

Checking the build with:

python infra/helper.py check_build --sanitizer address --engine libfuzzer --architecture i386 qt

then passes with a contradictory message:

ERROR: The size of coverage PC tables does not match the number of instrumented PCs. This might be a compiler bug, please contact the libFuzzer developers. Also check https://bugs.llvm.org/show_bug.cgi?id=34636 for possible workarounds (tl;dr: don't use the old GNU ld)

14 fuzzers total, 1 seem to be broken (7.142857142857143%). Check build passed.

Shouldn't check_build fail and report this?

If I should provide any more details, please let me know.

Cheers, Robert

https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=26560 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=30197 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=33364 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=33438 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=33678 https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=33684

evverx commented 3 years ago

Shouldn't check_build fail and report this?

I think it depends. I'm not sure if it helps but It has been kind of configurable since 2361c62433d840 was merged. CIFuzz supports allowed-broken-targets-percentage as well: https://google.github.io/oss-fuzz/getting-started/continuous-integration/#optional-configuration

rlohning commented 3 years ago

I see how I could use this for my local run or for CIFuzz, but how could I set the percentage to zero for the tests in oss-fuzz?

I'll have to believe you that there are cases when one wants to allow a broken percentage, but shouldn't oss-fuzz in this case make sure that issues opened for the broken fuzzer are not being closed while the fuzzer is unable to run?

evverx commented 3 years ago

I see how I could use this for my local run or for CIFuzz, but how could I set the percentage to zero for the tests in oss-fuzz?

I don't think it's possible. Though I agree If neither CIFuzz nor any other CI script running infra/helper.py is used it would probably make sense to allow setting allowed-broken-targets-percentage on OSS-Fuzz.

shouldn't oss-fuzz in this case make sure that issues opened for the broken fuzzer are not being closed while the fuzzer is unable to run?

I think it probably should but it seems to be complicated in the sense it's not always obvious how to tell crashes like that from fuzz targets failing due to "shallow" issues that haven't been fixed yet.