search

rust-fuzz / libfuzzer

Rust bindings and utilities for LLVM’s libFuzzer
Apache License 2.0
215 stars 46 forks source link

Update libFuzzer to llvm/llvm-project@60e32a1 #89

Closed fitzgen closed 2 years ago

fitzgen commented 2 years ago

Supercedes #88

nagisa commented 2 years ago

bors r+

never remember if we have this set up here or not... (we also will need to update our test suite sometime down the line for new pass names I bet)

bors[bot] commented 2 years ago

Configuration problem: bors.toml: not found

fitzgen commented 2 years ago

Need to switch -Cpasses=sancov to -Cpasses=sancov-module for newer rustc/LLVM. New commit should fix CI.

fitzgen commented 2 years ago

Linking issues in CI. Not totally sure what's going on here.

https://github.com/rust-fuzz/libfuzzer/runs/4930210184?check_suite_focus=true#step:5:74

error: linking with `cc` failed: exit status: 1
  |
  = note: "cc" "-m64" "-Wl,-Bstatic" "-Wl,--whole-archive" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc-nightly_rt.asan.a" "-Wl,--no-whole-archive" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.0.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.1.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.10.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.11.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.12.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.13.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.14.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.15.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.2.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.3.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.4.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.5.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.6.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.7.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.8.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.9.rcgu.o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.1w1yy0cu9bxvqmzw.rcgu.o" "-Wl,--as-needed" "-L" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps" "-L" "/home/runner/work/libfuzzer/libfuzzer/target/release/build/libfuzzer-sys-60bdc856fc564d33/out" "-L" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/liblibfuzzer_sys-9667746bd1260caf.rlib" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/libonce_cell-7957dbfbd59bdd9d.rlib" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/libarbitrary-6173795e953c05ad.rlib" "-Wl,--start-group" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-516357af627e1a7d.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-9cb5ef8235cf10de.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-c94e3d0069d19330.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-5a532766d1e80b5b.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-d08f0c83526179dc.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-d995c6cafdb3d253.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-02ebc04a72bd9433.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-5cbc32ae14534ca9.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-969ebc8623abfcc9.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-7f98f837d3579544.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-c79b77d9c2e73c7a.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-c7b2792f3bc0d0eb.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-7558dde8464d46ce.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-b3d84848ab790cf7.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-f48e57d6d73020ee.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-ffaac76756e3f83b.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-2a6a2797f7a73818.rlib" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-0e3656b1fda5fd7b.rlib" "-Wl,--end-group" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-f98cce041f593917.rlib" "-Wl,-Bdynamic" "-lstdc++" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-znoexecstack" "-L" "/home/runner/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro,-znow" "-Wl,-O1" "-nodefaultlibs"
  = note: `.text.sancov.module_ctor_8bit_counters.9' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.9]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.3.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.9[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.3.rcgu.o
          `.text.sancov.module_ctor_8bit_counters.6' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.6]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.1.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.6[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.1.rcgu.o
          `.text.sancov.module_ctor_8bit_counters.4' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.4]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.10.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.4[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.10.rcgu.o
          `.text.sancov.module_ctor_8bit_counters.8' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.8]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.15.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.8[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.15.rcgu.o
          `.text.sancov.module_ctor_8bit_counters.11' referenced in section `.init_array.2[sancov.module_ctor_8bit_counters.11]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.2.rcgu.o: defined in discarded section `.text.sancov.module_ctor_8bit_counters.11[sancov.module_ctor_8bit_counters]' of /home/runner/work/libfuzzer/libfuzzer/target/release/deps/example-5f8d6a2d2829c12b.example.ad2a7976-cgu.2.rcgu.o
          collect2: error: ld returned 1 exit status
Badel2 commented 2 years ago

@fitzgen I get the same error when trying to compile master. A workaround is to add -Ccodegen-units=1 \ to every command in ci/script.sh. But note that this will probably force end users to also add this flag when compiling.

fitzgen commented 2 years ago

Which means we would probably need to add it to cargo-fuzz as well.

And that is going to slow down build times too. Very much not ideal. Would prefer to figure out how to fix these linker errors without resorting to a single codegen unit, but I don't have time to dig into this myself right now.

Badel2 commented 2 years ago

I tried to take a look but linker errors are a bit outside my area of expertise. But I did found another strange bug, see #90.

fitzgen commented 2 years ago

Okay I pushed a commit to do -Ccodegen-units=1.

This is not ideal, but it does provide two things:

  1. It fixes bizarre linker errors about missing sancov symbols.

  2. It allows LLVM to do inlining that it otherwise refuses to do. For some reason, when sanitizers are enabled, LLVM refuses to inline across codegen units. This is a problem because trivial methods like Vec::len won't be inlined, resulting in 100x slowdowns.

cargo fuzz already restricts its builds to a single codegen unit, so we might as well do the same thing in CI here.

fitzgen commented 2 years ago

Success!

alexcrichton commented 2 years ago

FWIW there's some more info about the 1 cgu issue at https://github.com/rust-fuzz/cargo-fuzz/pull/215