Compilation failures since 0.3.3 due to missing header

mhlakhani commented 3 years ago

We use this crate to fuzz libra (https://github.com/libra/libra) - and it's been working really well, thank you for this integration!

We tried upgrading to 0.3.3 a while back and had to revert, because the build was breaking on certain machines which don't have LLVM source on the include path. I am not sure how to fix this, but I wanted to file an issue in the hopes someone can hepl fix it.

As far as I can tell, this is due to an upstream issue. 0.3.3 includes an updated version of libfuzzer (https://github.com/rust-fuzz/libfuzzer/commit/0493bb89c2e97d1dac8a3b52dd2c951d50310b5a) which most specifically includes https://github.com/llvm/llvm-project/commit/45482367eaf3e7aa6e00b667bf83ce700b390918

This includes a file that's not in the libfuzzer source tree, and I don't see this file being present in this source repo.

I'm not super familiar with how the build.rs scripts work under the hood, but it seems like we need to ensure that this file is present in the repo and appropriately passed to the include paths for the compiler.

If it's helpful, here are the logs from our system:

OPT_LEVEL = Some("3")
HOST = Some("x86_64-unknown-linux-gnu")
CXX_x86_64-unknown-linux-gnu = None
CXX_x86_64_unknown_linux_gnu = None
HOST_CXX = None
CXX = None
CXXFLAGS_x86_64-unknown-linux-gnu = None
CXXFLAGS_x86_64_unknown_linux_gnu = None
HOST_CXXFLAGS = None
CXXFLAGS = None
CRATE_CC_NO_DEFAULTS = None
DEBUG = Some("true")
CARGO_CFG_TARGET_FEATURE = Some("fxsr,mmx,sse,sse2")
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerCrossOver.o" "-c" "libfuzzer/FuzzerCrossOver.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerDataFlowTrace.o" "-c" "libfuzzer/FuzzerDataFlowTrace.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerDriver.o" "-c" "libfuzzer/FuzzerDriver.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerExtFunctionsDlsym.o" "-c" "libfuzzer/FuzzerExtFunctionsDlsym.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerExtFunctionsWeak.o" "-c" "libfuzzer/FuzzerExtFunctionsWeak.cpp"
exit code: 0
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerExtFunctionsWindows.o" "-c" "libfuzzer/FuzzerExtFunctionsWindows.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerExtraCounters.o" "-c" "libfuzzer/FuzzerExtraCounters.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerFork.o" "-c" "libfuzzer/FuzzerFork.cpp"
exit code: 0
exit code: 0
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerIO.o" "-c" "libfuzzer/FuzzerIO.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerIOPosix.o" "-c" "libfuzzer/FuzzerIOPosix.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerIOWindows.o" "-c" "libfuzzer/FuzzerIOWindows.cpp"
exit code: 0
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerInterceptors.o" "-c" "libfuzzer/FuzzerInterceptors.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerLoop.o" "-c" "libfuzzer/FuzzerLoop.cpp"
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerMain.o" "-c" "libfuzzer/FuzzerMain.cpp"
exit code: 0
cargo:warning=libfuzzer/FuzzerInterceptors.cpp:30:45: fatal error: sanitizer/common_interface_defs.h: No such file or directory
cargo:warning= #include <sanitizer/common_interface_defs.h>
cargo:warning=                                             ^
cargo:warning=compilation terminated.
exit code: 1
running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerMerge.o" "-c" "libfuzzer/FuzzerMerge.cpp"
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
exit code: 0
--- stderr
error occurred: Command "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-m64" "-Wall" "-Wextra" "-std=c++11" "-fno-omit-frame-pointer" "-w" "-o" "/sanitized_root/target/x86_64-unknown-linux-gnu/release/build/libfuzzer-sys-f9d0e2c655ba9127/out/libfuzzer/FuzzerInterceptors.o" "-c" "libfuzzer/FuzzerInterceptors.cpp" with args "c++" did not execute successfully (status code exit code: 1).

alex commented 3 years ago

Hmm, the most relevant thing is maybe https://github.com/rust-fuzz/libfuzzer/blob/master/update-libfuzzer.sh

As you can see we check out the llvm repo and just copy the libfuzzer dir.

I could have sworn the libfuzzer docs used to claim you could just build it with cc *.c but I don't see that in the docs any longer.

mhlakhani commented 3 years ago

@alex my understanding of that script is that it will copy over the files from upstream, but doesn't copy over that additional header. Fixing that wouldn't be sufficient (I think) since we also need to update https://github.com/rust-fuzz/libfuzzer/blob/master/build.rs to somehow get this on the path.

I could have sworn the libfuzzer docs used to claim you could just build it with cc *.c but I don't see that in the docs any longer.

I also recall seeing that, but I can't find it anymore - and I think https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/fuzzer/build.sh also is proof of that assumption.

I am happy to report this on the libfuzzer mailing list instead if you think that's a better place for this fix.

alex commented 3 years ago

I think filing a bug about build.sh being broken is a reasonable thing -- but we shoul also fix this package.

On Wed, Aug 26, 2020 at 7:52 PM Hasnain Lakhani notifications@github.com wrote:

@alex https://github.com/alex my understanding of that script is that it will copy over the files from upstream, but doesn't copy over that additional header. Fixing that wouldn't be sufficient (I think) since we also need to update https://github.com/rust-fuzz/libfuzzer/blob/master/build.rs to somehow get this on the path.

I could have sworn the libfuzzer docs used to claim you could just build it with cc *.c but I don't see that in the docs any longer.

I also recall seeing that, but I can't find it anymore - and I think https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/fuzzer/build.sh also is proof of that assumption.

I am happy to report this on the libfuzzer mailing list instead if you think that's a better place for this fix.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rust-fuzz/libfuzzer/issues/65#issuecomment-681181661, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAAAGBARCFG2E5B4KPMKTXDSCWN35ANCNFSM4QMKFHCQ .

-- All that is necessary for evil to succeed is for good people to do nothing.

alex commented 3 years ago

This is getting fixed upstream: https://reviews.llvm.org/D86981

Once that's landed we can pull in the changes and do a release if we want