search

google / oss-fuzz

OSS-Fuzz - continuous fuzzing for open source software.
https://google.github.io/oss-fuzz
Apache License 2.0
10.47k stars 2.22k forks source link

Better support for fuzzing Rust, remaining - code coverage; MSan; LSan in ASan #3469

Closed Dor1s closed 1 year ago

Dor1s commented 4 years ago

Tracking bug for ideas / suggestions

jonathanmetzman commented 4 years ago

Fixing this inconsistency/getting the normal thing working on jenkins: https://github.com/google/oss-fuzz/pull/3298

jfoote commented 4 years ago

IIUC https://github.com/rust-lang/rust/issues/34701 could be a precursor to adding support for something like clang source-based coverage for Rust targets (related issue: https://github.com/google/oss-fuzz/issues/3468).

catenacyber commented 4 years ago

Support MSAN when mixing C and Rust cf https://github.com/google/oss-fuzz/issues/2145#issuecomment-485781098

inferno-chromium commented 4 years ago

Didnt realise sanitizers were disabled for Rust projects, this definitely seems bad. https://github.com/google/oss-fuzz/blob/44f4b89366ca38840cd04e83bf33b8eaa52ab6c5/projects/mp4parse-rust/build.sh#L24

inferno-chromium commented 4 years ago

https://github.com/google/oss-fuzz/pull/3830 simplifies rust workflow, adds missing sanitizer in many projects. migrated most in https://github.com/google/oss-fuzz/pull/3840, libra is remaining.

inferno-chromium commented 4 years ago

@catenacyber - MSan should work after #3830, maybe you want to try with some projects ? If you can also find what things are missing, that would be much appreciated.

catenacyber commented 4 years ago

Interesting, now I get a build failure ( bug found by MSAN ?) :

RUSTUP_HOME=/rust/rustup \
    CARGO_HOME="/rust" \
    CARGO_TARGET_DIR="/src/suricata/rust/target" \
    /rust/bin/cargo build --release \
        --features " " 
   Compiling autocfg v1.0.0
   Compiling cfg-if v0.1.9
   Compiling semver-parser v0.7.0
   Compiling libc v0.2.70
   Compiling arrayvec v0.4.12
   Compiling bitflags v1.2.1
   Compiling ryu v1.0.4
   Compiling memchr v2.3.3
   Compiling version_check v0.9.1
   Compiling nodrop v0.1.14
   Compiling getrandom v0.1.14
   Compiling static_assertions v0.3.4
   Compiling ppv-lite86 v0.2.8
   Compiling siphasher v0.3.3
   Compiling proc-macro2 v0.4.30
   Compiling unicode-xid v0.1.0
   Compiling syn v0.15.44
   Compiling build_const v0.2.1
   Compiling byteorder v1.3.4
   Compiling num-derive v0.2.5
   Compiling base64 v0.11.0
   Compiling widestring v0.4.0
   Compiling semver v0.9.0
   Compiling num-traits v0.2.11
   Compiling num-integer v0.1.42
   Compiling num-bigint v0.2.6
   Compiling num-rational v0.2.4
   Compiling num-iter v0.1.40
   Compiling num-complex v0.2.4
error: failed to run custom build command for `libc v0.2.70`

Caused by:
  process didn't exit successfully: `/src/suricata/rust/target/release/build/libc-774c2aec654d22a9/build-script-build` (exit code: 77)
--- stderr
Uninitialized bytes in __interceptor_memchr at offset 0 inside [0x701000000000, 4)
==19952==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55f8b392f2ee in std::sys::unix::memchr::memchr::hd921ac10d14dc82d /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/sys/unix/memchr.rs:6:9
    #1 0x55f8b392f2ee in std::memchr::memchr::h75af9c209e288688 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/memchr.rs:25:5
    #2 0x55f8b392f2ee in std::ffi::c_str::CString::_new::hd3fc15f3d02971b6 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/ffi/c_str.rs:355:15
    #3 0x55f8b392f2ee in std::ffi::c_str::CString::new::h2da21344d9607d80 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/ffi/c_str.rs:351:9
    #4 0x55f8b392f2ee in std::thread::Thread::new::_$u7b$$u7b$closure$u7d$$u7d$::h957656e8072d73d0 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/thread/mod.rs:1143:26
    #5 0x55f8b392f2ee in core::option::Option$LT$T$GT$::map::h5c5d6edf546fcd7f /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libcore/option.rs:458:29
    #6 0x55f8b392f2ee in std::thread::Thread::new::h5dfa556c4c696141 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/thread/mod.rs:1143:13
    #7 0x55f8b3938194 in std::rt::lang_start_internal::h9a98dc8e2c22711e /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/rt.rs:44:22
    #8 0x55f8b3927beb in main (/src/suricata/rust/target/release/build/libc-774c2aec654d22a9/build-script-build+0x6fbeb)
    #9 0x7f0ac815082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #10 0x55f8b38c7478 in _start (/src/suricata/rust/target/release/build/libc-774c2aec654d22a9/build-script-build+0xf478)

SUMMARY: MemorySanitizer: use-of-uninitialized-value /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/sys/unix/memchr.rs:6:9 in std::sys::unix::memchr::memchr::hd921ac10d14dc82d

Adding export MSAN_OPTIONS="halt_on_error=0:exitcode=0:report_umrs=0" gives then 

   Compiling rand_core v0.5.1
error: couldn't read /src/suricata/rust/target/release/build/crc-43db1b02c9ee945c/out/crc16_constants.rs: No such file or directory (os error 2)
 --> /rust/registry/src/github.com-1ecc6299db9ec823/crc-1.8.1/src/crc16.rs:8:1
  |
8 | include!(concat!(env!("OUT_DIR"), "/crc16_constants.rs"));
  | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
  = note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)

error: aborting due to previous error

error: could not compile `crc`.
alexcrichton commented 4 years ago

Ah the problem with that @catenacyber is that the usage of RUSTFLAGS is also affecting build scripts, which means sanitizers are running as part of the build (unfortunately). That can be fixed with a --target flag, however.

catenacyber commented 4 years ago

Thanks @alexcrichton I am not sure I understand what you mean. Where should I add the --target flag ? to the cargo build command ? With which value ?

More info, when trying to build with ASAN, I get error: /tmp/cargo-installDd2VEJ/release/deps/libserde_derive-e8fab3128f048517.so: undefined symbol: __asan_stack_free_7

alexcrichton commented 4 years ago

Yes you can add --target to cargo build, and you'll probably want --target x86_64-unknown-linux-gnu. That second error you get is because procedural macros are trying to get built with sanitizers, which doesn't work.

catenacyber commented 4 years ago

Thanks @alexcrichton Using export CARGO_BUILD_TARGET="x86_64-unknown-linux-gnu" made it work.

Then @inferno-chromium it seems I am back to the previous problem of the fuzz target mixing C and Rust : I get 

==11==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0xd71b43 in core::result::Result$LT$T$C$E$GT$::unwrap::h94893129de2c63ee /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libcore/result.rs:1004:13
    #1 0xd71b43 in rs_dns_udp_register_parser /src/suricata/rust/src/dns/dns.rs:993:24

with the line being let default_port = std::ffi::CString::new("[53]").unwrap();

inferno-chromium commented 4 years ago

@catenacyber - i have fixed this now in https://github.com/google/oss-fuzz/pull/3859 , base build images should be rebuilt in one hour, then you need to infra/helper.py pull_images and retry.

inferno-chromium commented 4 years ago

Ok for MSan, check_build fails (which actually runs the binaries). @alexcrichton , can you please check below. Many projects use cargo fuzz build, is there a difference.

python infra/helper.py pull_images
python infra/helper.py build_fuzzers --sanitizer memory mp4parse-rust
python infra/helper.py check_build --sanitizer memory mp4parse-rust
Running: docker run --rm --privileged -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=memory -e ARCHITECTURE=x86_64 -v /build/oss-fuzz/build/out/mp4parse-rust:/out -t gcr.io/oss-fuzz-base/base-runner test_all
INFO: performing bad build checks for /tmp/not-out/avif.
INFO: performing bad build checks for /tmp/not-out/mp4.
Broken fuzz targets (2):
avif:
BAD BUILD: /tmp/not-out/avif seems to have either startup crash or exit:
INFO: Seed: 1337
INFO: Loaded 1 modules   (25295 inline 8-bit counters): 25295 [0x55eb5fd167a8, 0x55eb5fd1ca77), 
INFO: Loaded 1 PC tables (25295 PCs): 25295 [0x55eb5fd1ca78,0x55eb5fd7f768), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
==78==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x55eb5f4fa87f in rust_fuzzer_test_input /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20
    #1 0x55eb5f71173f in __rust_try (/tmp/not-out/avif+0x36873f)
    #2 0x55eb5f711324 in std::panicking::try::hfc14f47abd791d38 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panicking.rs:274:15
    #3 0x55eb5f711324 in std::panic::catch_unwind::h3c37ac4d75f6cfa4 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panic.rs:394:14
    #4 0x55eb5f711324 in LLVMFuzzerTestOneInput /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:25:22
    #5 0x55eb5f786a4d in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:556:15
    #6 0x55eb5f78af58 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:743:3
    #7 0x55eb5f78c71e in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:794:3
    #8 0x55eb5f726f28 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:826:6
    #9 0x55eb5f70f950 in main /src/libfuzzer/FuzzerMain.cpp:19:10
    #10 0x7fcf97fc382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #11 0x55eb5f473398 in _start (/tmp/not-out/avif+0xca398)

DEDUP_TOKEN: rust_fuzzer_test_input--__rust_try--std::panicking::try::hfc14f47abd791d38
SUMMARY: MemorySanitizer: use-of-uninitialized-value /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20 in rust_fuzzer_test_input
  ORIGIN: invalid (0). Might be a bug in MemorySanitizer origin tracking.
    This could still be a bug in your code, too!
Unique heap origins: 35
Stack depot allocated bytes: 2376
Unique origin histories: 10
History depot allocated bytes: 240
Exiting
MS: 0 ; base unit: 0000000000000000000000000000000000000000

artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64: 
mp4:
BAD BUILD: /tmp/not-out/mp4 seems to have either startup crash or exit:
INFO: Seed: 1337
INFO: Loaded 1 modules   (25344 inline 8-bit counters): 25344 [0x556b13d2f7d8, 0x556b13d35ad8), 
INFO: Loaded 1 PC tables (25344 PCs): 25344 [0x556b13d35ad8,0x556b13d98ad8), 
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
==82==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x556b135130dc in rust_fuzzer_test_input /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20
    #1 0x556b13729f9f in __rust_try (/tmp/not-out/mp4+0x369f9f)
    #2 0x556b13729b84 in std::panicking::try::hfc14f47abd791d38 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panicking.rs:274:15
    #3 0x556b13729b84 in std::panic::catch_unwind::h3c37ac4d75f6cfa4 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panic.rs:394:14
    #4 0x556b13729b84 in LLVMFuzzerTestOneInput /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:25:22
    #5 0x556b1379f2ad in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:556:15
    #6 0x556b137a37b8 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:743:3
    #7 0x556b137a4f7e in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:794:3
    #8 0x556b1373f788 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:826:6
    #9 0x556b137281b0 in main /src/libfuzzer/FuzzerMain.cpp:19:10
    #10 0x7f608ce0782f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
    #11 0x556b1348ac18 in _start (/tmp/not-out/mp4+0xcac18)

DEDUP_TOKEN: rust_fuzzer_test_input--__rust_try--std::panicking::try::hfc14f47abd791d38
SUMMARY: MemorySanitizer: use-of-uninitialized-value /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20 in rust_fuzzer_test_input
  ORIGIN: invalid (0). Might be a bug in MemorySanitizer origin tracking.
    This could still be a bug in your code, too!
Unique heap origins: 35
Stack depot allocated bytes: 2392
Unique origin histories: 10
History depot allocated bytes: 240
Exiting
MS: 0 ; base unit: 0000000000000000000000000000000000000000

artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64: 
ERROR: 100% of fuzz targets seem to be broken. See the list above for a detailed information.
Check build failed.
alexcrichton commented 4 years ago

Ah sorry I don't think I'm knowledgeable enough about fuzzers/msan/etc to understand what's going on there. Given "Might be a bug in MemorySanitizer origin tracking." it looks like something possibly wasn't compiled correctly? I don't have much experience myself in using other fuzzers in Rust. It might be worthwhile to raise this on the cargo-fuzz repo and see if others can help out?

inferno-chromium commented 4 years ago

@catenacyber @victorjulien, just fyi suricata project build seems broken due to recent default all Rust flags [https://github.com/google/oss-fuzz/pull/3859]. You would need to remove stable cargo from Dockerfile, use rust nightly directly(no need to install) and add this target CARGO_BUILD_TARGET="x86_64-unknown-linux-gnu" to make it work with sanitizer. I could not go far beyond that, there is some weird error - "make[2]: *** No rule to make target '../rust/target/release/libsuricata.a', needed by 'suricata'. Stop." python butler/infra.py pull_images python butler/infra.py build_images suricata

catenacyber commented 4 years ago

@inferno-chromium I will take a good look at it next week The "weird error" is that Suricata build system is seeking the rust static library in some path which is not compatible with the use of CARGO_BUILD_TARGET

Quick and dirty patch I have been using is :

diff --git a/configure.ac b/configure.ac
index 127848234..747b96afb 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2487,7 +2487,11 @@ fi
     if test "x$cross_compiling" = "xyes"; then
       RUST_SURICATA_LIB_XC_DIR="${host_alias}/"
     else
-      RUST_SURICATA_LIB_XC_DIR=
+      if test "x$CARGO_BUILD_TARGET" = "x"; then
+        RUST_SURICATA_LIB_XC_DIR=
+      else
+        RUST_SURICATA_LIB_XC_DIR="${CARGO_BUILD_TARGET}/"
+      fi
     fi

     if test "x$enable_debug" = "xyes"; then

I will try to adding export cross_compiling=yes in build.sh

catenacyber commented 4 years ago

Quick and dirty patch seems good in fact https://github.com/OISF/suricata/pull/4986 Another PR to oss-fuzz is coming as well

inferno-chromium commented 4 years ago

Fyi, msan can't be enabled due to https://github.com/google/oss-fuzz/pull/3988#issuecomment-644230600

catenacyber commented 4 years ago

Is there a current work to have MSAN with recompiled std library ?

inferno-chromium commented 4 years ago

Is there a current work to have MSAN with recompiled std library ?

I dont know what is required here or changes needed in cargo fuzz.

tmiasko commented 4 years ago

For simple cargo projects the minimal set of flags required for MemorySanitizer would be:

$ export CC=clang CFLAGS=-fsanitize=memory RUSTFLAGS=-Zsanitizer=memory RUSTDOCFLAGS=-Zsanitizer=memory
$ cargo test --workspace -Z build-std --target x86_64-unknown-linux-gnu

For projects using cargo-fuzz it would be additionally:

$ export CUSTOM_LIBFUZZER_PATH=...
$ cargo fuzz run fuzz_target -s memory -Z build-std --target x86_64-unknown-linux-gnu
catenacyber commented 4 years ago

Thanks @tmiasko it looks like it works cf https://github.com/OISF/suricata/pull/5395

jrmuizel commented 3 years ago

rust-lang/rust#34701 is merged now which should help with getting coverage information.

fitzgen commented 3 years ago

FWIW, cargo fuzz version 0.10.0 now has a cargo fuzz coverage command that will build the fuzz target(s) with coverage instrumentation and then run the fuzz target(s) over the corpus and generate a merged coverage data file for the corpus.

https://rust-fuzz.github.io/book/cargo-fuzz/coverage.html

catenacyber commented 3 years ago

Thanks for this news @fitzgen Does it handle source remaps cf https://github.com/google/oss-fuzz/pull/5392 And removing optimization cf https://github.com/rust-lang/rust/issues/82144 ? Could this be cargo fuzz -s coverage as another sanitizer ?

fitzgen commented 3 years ago

Does it handle source remaps cf #5392

It does not. Is this something every user will need, or just oss-fuzz style users that move the binaries to new locations/machines? It hasn't been necessary for any of the local coverage experiments I've performed yet.

And removing optimization cf rust-lang/rust#82144 ?

Optimizations can be controlled via passing the -O/--release flag, and is orthogonal from coverage builds.

Agreed that the upstream rustc bug should get fixed, but I'm not convinced it warrants entangling optimized vs not builds together with coverage vs not builds.

Could this be cargo fuzz -s coverage as another sanitizer ?

Do you want the ability to build coverage binaries without automatically running them over the corpus to produce coverage data? We can definitely make it so that you can build a coverage-instrumented binary with cargo fuzz build --coverage.

catenacyber commented 3 years ago

Does it handle source remaps cf #5392

It does not. Is this something every user will need, or just oss-fuzz style users that move the binaries to new locations/machines?

It is not about a new machine (this is handled by llvm-cov flags) This is rather about having absolute file paths cf https://github.com/rust-lang/cargo/issues/5450

It hasn't been necessary for any of the local coverage experiments I've performed yet.

You can try wasmtime from oss-fuzz You can try the regalloc repository, running it from the fuzz directory

And removing optimization cf rust-lang/rust#82144 ?

Optimizations can be controlled via passing the -O/--release flag, and is orthogonal from coverage builds.

Agreed that the upstream rustc bug should get fixed, but I'm not convinced it warrants entangling optimized vs not builds together with coverage vs not builds.

It may be better to wait for the rustc bug fix...

Could this be cargo fuzz -s coverage as another sanitizer ?

Do you want the ability to build coverage binaries without automatically running them over the corpus to produce coverage data?

Yes :-)

We can definitely make it so that you can build a coverage-instrumented binary with cargo fuzz build --coverage.

For oss-fuzz projects, it could be easier with cargo fuzz build -s coverage as we can use cargo fuzz build -s $SANITIZER with SANITIZER being an environment variable that can be address, memory, coverage...