Evaluate quality of reports against oss-fuzz projects

[oliverchang]

The reports for the OSS-Fuzz projects where the introspector successfully runs are now public at:

https://oss-fuzz-introspector.storage.googleapis.com/

We should go and evaluate these to ensure their quality, before we make them more broadly available to users.

e.g. I browsed a few and found some issues/weirdness:

• arduinojson has weird fuzzer names (e.g. "srcarduinojsonextrasfuzzingjson_fuzzer.cpp")

• libxml2 has no functions hit data for xml.c.

[Navidem]

These projects fail to generate fuzz_report.html report while the build is successful:

abseil-cpp
alembic
bitcoin-core
boringssl
casync
cel-cpp
circl
clamav
dart
dav1d
double-conversion
draco
fribidi
grpc-httpjson-transcoding
harfbuzz
http-pattern-matcher
jbig2dec
libavc
libcacard
libcoap
libhevc
libjxl
librawspeed
libspectre
libssh
libtpms
lwan
open62541
opencensus-cpp
openexr
opensc
ots
perfetto
pffft
s2opc
spice-usbredir
systemd
tarantool
tcmalloc
tidy-html5
tink
tmux
tpm2-tss
upb

A quick check in couple of logs shows: INFO:fuzz_data_loader: - found 0 profiles to load

[Navidem]

A quick check in couple of logs shows: INFO:fuzz_data_loader: - found 0 profiles to load

For the ones encountering 0 profiles to load this can be the culprit: introspector is storing fuzzerLogFile*.data and fuzzerLogFile*.data.yaml in /work while compile script expects them in /src

This is the case for lwan.

[Navidem]

These projects fail to generate fuzz_report.html report while the build is successful:

52 helped to reduce this number to 19 projects:

abseil-cpp
bitcoin-core
cel-cpp
circl
clamav
dart
grpc-httpjson-transcoding
http-pattern-matcher
libcoap
libspectre
libssh
opencensus-cpp
opensc
tarantool
tcmalloc
tink
tmux
tpm2-tss
upb

checking abseil-cpp opencensus-cpp tcmalloc shows surprisingly this log message: Fuzz introspector is not running while the env var is correctly set. abseil-cpp shows this behavior on local instance too.

[Navidem]

Quick update: Had to set FUZZ_INTROSPECTOR at Dockerfile otherwise bazel builds could not see the environment variable. This fixed abseil-cpp.

There is a correct way of passing environment variables to bazel builds: https://github.com/google/oss-fuzz/pull/7367 This PR fixes Fuzz introspector is not running cases.

[Navidem]

Besides the ones that skip introspector pass because of another main() (#66), we have the following 8 projects that the introspector pass is not run at all:

bitcoin-core
circl
clamav
dart
libcoap
opensc
tink
tpm2-tss

[DavidKorczynski]

bitcoin-core

Note bitcoin-core has a couple of issues on fuzz-introspector list with details on this. Additionally, they use a neat hack of only compiling a single executable and then substituting the string of a function name which will be the relevant fuzzer entrypoint in the resulting binary, as a way to produce many targets without compiling all targets from scratch -- fuzz-introspector cannot deal with that atm (nor likely in the future). See details here: https://github.com/ossf/fuzz-introspector/issues/44#issuecomment-1048517052

[Navidem]

Note bitcoin-core has a couple of issues on fuzz-introspector list with details on this. Additionally, they use a neat hack of only compiling a single executable and then substituting the string of a function name which will be the relevant fuzzer entrypoint in the resulting binary, as a way to produce many targets without compiling all targets from scratch -- fuzz-introspector cannot deal with that atm (nor likely in the future). See details here: #44 (comment)

Ack on bitcoin-core, the rest worth more investigation as it looks like the fuzz introspector pass is being skipped silently.