search

ossf / fuzz-introspector

Fuzz Introspector -- introspect, extend and optimise fuzzers
https://fuzz-introspector.readthedocs.io
Apache License 2.0
385 stars 57 forks source link

[OSS-Fuzz] Projects fail even after using old pass manager #322

Open Navidem opened 2 years ago

Navidem commented 2 years ago

There are the projects still failing (residue from #305):

cyclonedds
dart
git
gnutls
keystone
libcoap
libfido2
libspectre
libvips
nss
opencv
opensc
wolfssl

They were being successfully built before merging https://github.com/google/oss-fuzz/pull/7788, and https://github.com/google/oss-fuzz/pull/7828 did not help to fix them (failing with different kinds of error now).

oliverchang commented 2 years ago

@AdamKorcz @DavidKorczynski thoughts?

Some snippets:

https://oss-fuzz-build-logs.storage.googleapis.com/log-f76d4e66-6430-4815-8fd9-8a93fc13aa70.txt

Step #7 - "compile-libfuzzer-introspector-x86_64": make: *** [Makefile:3588: fuzz-pack-idx] Error 1
Step #7 - "compile-libfuzzer-introspector-x86_64": make: *** Waiting for unfinished jobs....
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-e3f060.o:ld-temp.o:function parse_archive_args: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-e3f060.o:ld-temp.o:function create_branches_recursively: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-e3f060.o:ld-temp.o:function dwim_branch_start: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-e3f060.o:ld-temp.o:function setup_tracking: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
Step #7 - "compile-libfuzzer-introspector-x86_64": make: *** [Makefile:3588: fuzz-pack-headers] Error 1
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-4ec2f3.o:ld-temp.o:function parse_archive_args: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-4ec2f3.o:ld-temp.o:function create_branches_recursively: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-4ec2f3.o:ld-temp.o:function dwim_branch_start: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-4ec2f3.o:ld-temp.o:function setup_tracking: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

A bunch also have: (e.g. https://oss-fuzz-build-logs.storage.googleapis.com/log-f5187e36-521b-4360-b875-f1e67a68f92c.txt)

Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:File: /workspace/out/libfuzzer-introspector-x86_64/pdu_parse_fuzzer is executable
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:File: /workspace/out/libfuzzer-introspector-x86_64/llvm-symbolizer is executable
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:File: /workspace/out/libfuzzer-introspector-x86_64/split_uri_fuzzer is executable
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Pairings: []
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Ending fuzz introspector post-processing
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Loading profiles
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 0 profiles to load
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Found no profiles. Exiting
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Ending fuzz introspector report generation
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Ending fuzz introspector post-processing

Some timeouts also (https://oss-fuzz-build-logs.storage.googleapis.com/log-3d9408b5-0dc0-4570-a994-e0991f962401.txt)

Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding LZW_GenerateStream
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding calcInitCodeLen
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding calcNextPower2Ex
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding cgif_raw_addframe
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding writeDummyBytes
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding initAppExtBlock
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding initMainHeader
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding cgif_raw_newgif
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-0-hqvsP8FCGD.data
TIMEOUT
ERROR: context deadline exceeded
DavidKorczynski commented 2 years ago

A bunch also have: (e.g. https://oss-fuzz-build-logs.storage.googleapis.com/log-f5187e36-521b-4360-b875-f1e67a68f92c.txt)

Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:File: /workspace/out/libfuzzer-introspector-x86_64/pdu_parse_fuzzer is executable
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:File: /workspace/out/libfuzzer-introspector-x86_64/llvm-symbolizer is executable
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_utils:File: /workspace/out/libfuzzer-introspector-x86_64/split_uri_fuzzer is executable
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Pairings: []
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Ending fuzz introspector post-processing
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:[+] Loading profiles
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 0 profiles to load
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Found no profiles. Exiting
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Ending fuzz introspector report generation
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Ending fuzz introspector post-processing

This is because the fuzzer is compiled without OSS-Fuzz flags, and in particular the LTO flags. See lines: 

Step #7 - "compile-libfuzzer-introspector-x86_64": clang -I../../include -I../../include -pedantic -Wall -Wcast-qual -Wextra -Wformat-security -Winline -Wmissing-declarations -Wmissing-prototypes -Wnested-externs -Wpointer-arith -Wshadow -Wstrict-prototypes -Wswitch-default -Wswitch-enum -Wunused -Wwrite-strings  -Wunused-result  -std=c99 -Wno-missing-prototypes -Wno-missing-declarations -c pdu_parse_target.c -o /workspace/out/libfuzzer-introspector-x86_64/pdu_parse_target.o
Step #7 - "compile-libfuzzer-introspector-x86_64": clang++ -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -flegacy-pass-manager -flto -fno-inline-functions -fuse-ld=gold -Wno-unused-command-line-argument -fsanitize=fuzzer-no-link -stdlib=libc++ -g -I../../include -I../../include   -O1 -fno-omit-frame-pointer -gline-tables-only -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -flegacy-pass-manager -flto -fno-inline-functions -fuse-ld=gold -Wno-unused-command-line-argument -fsanitize=fuzzer-no-link -stdlib=libc++ -g /workspace/out/libfuzzer-introspector-x86_64/pdu_parse_target.o ../../.libs/libcoap-3-notls.a  -lFuzzingEngine  -o /workspace/out/libfuzzer-introspector-x86_64/pdu_parse_fuzzer

The first line compiles the C stub for the fuzzer but does not include any of the OSS-Fuzz provided flags. This is needed for fuzz-introspector, because without LTO in the commandline it will be compiled to actual bytecode instead of llvm bytecode, where fuzz-introspector will not be able to find LLVMFuzzerTestOneInput and, therefore, declare there is no fuzzer. Looking at the makefile they hardcode the CFLAGS which are used in the compilation of the .c file, but the CXXFLAGS is used for linking which does include the oss-fuzz flags and hence why fuzz-introspector does indeed run.

I think it can be fixed by adding the CFLAGS from OSS-Fuzz on this line: https://github.com/obgm/libcoap/blob/4cad45319704b63f9534ef2d8a128af94bfe07bc/tests/oss-fuzz/Makefile.ci.in#L18

Did this really compile with introspector successfully before?

DavidKorczynski commented 2 years ago

Some timeouts also (https://oss-fuzz-build-logs.storage.googleapis.com/log-3d9408b5-0dc0-4570-a994-e0991f962401.txt)

Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding LZW_GenerateStream
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding calcInitCodeLen
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding calcNextPower2Ex
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding cgif_raw_addframe
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding writeDummyBytes
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding initAppExtBlock
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding initMainHeader
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader:Adding cgif_raw_newgif
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - loading /src/inspector/fuzzerLogFile-0-hqvsP8FCGD.data
TIMEOUT
ERROR: context deadline exceeded

I think this is simply just due to the added logic from the branch profiling.

The log is large so reducing that may help (I think @Navidem did this already?):

$ cat ./log-3d9408b5-0dc0-4570-a994-e0991f962401.txt | grep "We are in branch" | wc -l
248198
$ cat ./log-3d9408b5-0dc0-4570-a994-e0991f962401.txt | grep "No debug" | wc -l 
87156
$ cat log-3d9408b5-0dc0-4570-a994-e0991f962401.txt  | wc -l
397124

I think a way to speed up the branch profiler is merging the logic from here https://github.com/ossf/fuzz-introspector/blob/051932aa13b07f34e6c4844b084d3d928ae62138/llvm/lib/Transforms/FuzzIntrospector/FuzzIntrospector.cpp#L1145-L1147 which iterates through each basic block in each function into some other places in introspector where we do program iteration, e.g.: https://github.com/ossf/fuzz-introspector/blob/051932aa13b07f34e6c4844b084d3d928ae62138/llvm/lib/Transforms/FuzzIntrospector/FuzzIntrospector.cpp#L1033-L1044

There are other places where we can improve performance as well, e.g. https://github.com/ossf/fuzz-introspector/issues/195

DavidKorczynski commented 2 years ago 
Step #7 - "compile-libfuzzer-introspector-x86_64": make: *** [Makefile:3588: fuzz-pack-idx] Error 1
Step #7 - "compile-libfuzzer-introspector-x86_64": make: *** Waiting for unfinished jobs....
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-e3f060.o:ld-temp.o:function parse_archive_args: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-e3f060.o:ld-temp.o:function create_branches_recursively: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-e3f060.o:ld-temp.o:function dwim_branch_start: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-e3f060.o:ld-temp.o:function setup_tracking: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
Step #7 - "compile-libfuzzer-introspector-x86_64": make: *** [Makefile:3588: fuzz-pack-headers] Error 1
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-4ec2f3.o:ld-temp.o:function parse_archive_args: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-4ec2f3.o:ld-temp.o:function create_branches_recursively: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-4ec2f3.o:ld-temp.o:function dwim_branch_start: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": /tmp/lto-llvm-4ec2f3.o:ld-temp.o:function setup_tracking: error: undefined reference to 'common_exit'
Step #7 - "compile-libfuzzer-introspector-x86_64": clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

This is not because of introspector but because Git does not build atm: https://github.com/google/oss-fuzz/pull/7818

DavidKorczynski commented 2 years ago 
opencv
libfido2
wolfssl

are building correctly now

DavidKorczynski commented 2 years ago

cyclonedds failed because it's normal build was failing. once that was fixed introspector also builds correctly

Navidem commented 2 years ago

The log is large so reducing that may help (I think @Navidem did this already?):

Yes, the latest bump should have picked it up.

Navidem commented 2 years ago

Did this really compile with introspector successfully before?

For this case the build log is like this:

Screen Shot 2022-06-16 at 2 53 32 PM

The build was not failing, but it was not creating the fuzz_report.html with the silent message of 

Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 0 profiles to load
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Found no profiles. Exiting

So essentially there were no introspector report because the LTO was skipped. Somehow a change on introspector side caused the error be interpreted as build failure:

log from 06/15

Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 0 profiles to load
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Found no profiles. Exiting
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Ending fuzz introspector report generation
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Ending fuzz introspector post-processing
Step #7 - "compile-libfuzzer-introspector-x86_64": ********************************************************************************
Step #7 - "compile-libfuzzer-introspector-x86_64": Failed to build.
Step #7 - "compile-libfuzzer-introspector-x86_64": To reproduce, run:
Step #7 - "compile-libfuzzer-introspector-x86_64": python infra/helper.py build_image libcoap
Step #7 - "compile-libfuzzer-introspector-x86_64": python infra/helper.py build_fuzzers --sanitizer introspector --engine libfuzzer --architecture x86_64 libcoap
Step #7 - "compile-libfuzzer-introspector-x86_64": ********************************************************************************

vs log from 06/1

Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:fuzz_data_loader: - found 0 profiles to load
Step #7 - "compile-libfuzzer-introspector-x86_64": INFO:__main__:Found no profiles. Exiting
Finished Step #7 - "compile-libfuzzer-introspector-x86_64"

Anyways, this failure is not due to pass manager versioning.

DavidKorczynski commented 2 years ago

The reason it now shows as failed is because of https://github.com/ossf/fuzz-introspector/blob/e03cc475ab17e4e3fed01d5f169035cfd494db60/post-processing/main.py#L60-L62

Coming from this PR: https://github.com/ossf/fuzz-introspector/pull/256