Closed Dor1s closed 1 year ago
Fixing this inconsistency/getting the normal thing working on jenkins: https://github.com/google/oss-fuzz/pull/3298
IIUC https://github.com/rust-lang/rust/issues/34701 could be a precursor to adding support for something like clang source-based coverage for Rust targets (related issue: https://github.com/google/oss-fuzz/issues/3468).
Support MSAN when mixing C and Rust cf https://github.com/google/oss-fuzz/issues/2145#issuecomment-485781098
Didnt realise sanitizers were disabled for Rust projects, this definitely seems bad. https://github.com/google/oss-fuzz/blob/44f4b89366ca38840cd04e83bf33b8eaa52ab6c5/projects/mp4parse-rust/build.sh#L24
https://github.com/google/oss-fuzz/pull/3830 simplifies rust workflow, adds missing sanitizer in many projects. migrated most in https://github.com/google/oss-fuzz/pull/3840, libra is remaining.
@catenacyber - MSan should work after #3830, maybe you want to try with some projects ? If you can also find what things are missing, that would be much appreciated.
Interesting, now I get a build failure ( bug found by MSAN ?) :
RUSTUP_HOME=/rust/rustup \
CARGO_HOME="/rust" \
CARGO_TARGET_DIR="/src/suricata/rust/target" \
/rust/bin/cargo build --release \
--features " "
Compiling autocfg v1.0.0
Compiling cfg-if v0.1.9
Compiling semver-parser v0.7.0
Compiling libc v0.2.70
Compiling arrayvec v0.4.12
Compiling bitflags v1.2.1
Compiling ryu v1.0.4
Compiling memchr v2.3.3
Compiling version_check v0.9.1
Compiling nodrop v0.1.14
Compiling getrandom v0.1.14
Compiling static_assertions v0.3.4
Compiling ppv-lite86 v0.2.8
Compiling siphasher v0.3.3
Compiling proc-macro2 v0.4.30
Compiling unicode-xid v0.1.0
Compiling syn v0.15.44
Compiling build_const v0.2.1
Compiling byteorder v1.3.4
Compiling num-derive v0.2.5
Compiling base64 v0.11.0
Compiling widestring v0.4.0
Compiling semver v0.9.0
Compiling num-traits v0.2.11
Compiling num-integer v0.1.42
Compiling num-bigint v0.2.6
Compiling num-rational v0.2.4
Compiling num-iter v0.1.40
Compiling num-complex v0.2.4
error: failed to run custom build command for `libc v0.2.70`
Caused by:
process didn't exit successfully: `/src/suricata/rust/target/release/build/libc-774c2aec654d22a9/build-script-build` (exit code: 77)
--- stderr
Uninitialized bytes in __interceptor_memchr at offset 0 inside [0x701000000000, 4)
==19952==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x55f8b392f2ee in std::sys::unix::memchr::memchr::hd921ac10d14dc82d /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/sys/unix/memchr.rs:6:9
#1 0x55f8b392f2ee in std::memchr::memchr::h75af9c209e288688 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/memchr.rs:25:5
#2 0x55f8b392f2ee in std::ffi::c_str::CString::_new::hd3fc15f3d02971b6 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/ffi/c_str.rs:355:15
#3 0x55f8b392f2ee in std::ffi::c_str::CString::new::h2da21344d9607d80 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/ffi/c_str.rs:351:9
#4 0x55f8b392f2ee in std::thread::Thread::new::_$u7b$$u7b$closure$u7d$$u7d$::h957656e8072d73d0 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/thread/mod.rs:1143:26
#5 0x55f8b392f2ee in core::option::Option$LT$T$GT$::map::h5c5d6edf546fcd7f /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libcore/option.rs:458:29
#6 0x55f8b392f2ee in std::thread::Thread::new::h5dfa556c4c696141 /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/thread/mod.rs:1143:13
#7 0x55f8b3938194 in std::rt::lang_start_internal::h9a98dc8e2c22711e /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/rt.rs:44:22
#8 0x55f8b3927beb in main (/src/suricata/rust/target/release/build/libc-774c2aec654d22a9/build-script-build+0x6fbeb)
#9 0x7f0ac815082f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#10 0x55f8b38c7478 in _start (/src/suricata/rust/target/release/build/libc-774c2aec654d22a9/build-script-build+0xf478)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libstd/sys/unix/memchr.rs:6:9 in std::sys::unix::memchr::memchr::hd921ac10d14dc82d
Adding export MSAN_OPTIONS="halt_on_error=0:exitcode=0:report_umrs=0"
gives then
Compiling rand_core v0.5.1
error: couldn't read /src/suricata/rust/target/release/build/crc-43db1b02c9ee945c/out/crc16_constants.rs: No such file or directory (os error 2)
--> /rust/registry/src/github.com-1ecc6299db9ec823/crc-1.8.1/src/crc16.rs:8:1
|
8 | include!(concat!(env!("OUT_DIR"), "/crc16_constants.rs"));
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
= note: this error originates in a macro (in Nightly builds, run with -Z macro-backtrace for more info)
error: aborting due to previous error
error: could not compile `crc`.
Ah the problem with that @catenacyber is that the usage of RUSTFLAGS
is also affecting build scripts, which means sanitizers are running as part of the build (unfortunately). That can be fixed with a --target
flag, however.
Thanks @alexcrichton
I am not sure I understand what you mean.
Where should I add the --target
flag ? to the cargo build
command ? With which value ?
More info, when trying to build with ASAN, I get error: /tmp/cargo-installDd2VEJ/release/deps/libserde_derive-e8fab3128f048517.so: undefined symbol: __asan_stack_free_7
Yes you can add --target
to cargo build
, and you'll probably want --target x86_64-unknown-linux-gnu
. That second error you get is because procedural macros are trying to get built with sanitizers, which doesn't work.
Thanks @alexcrichton
Using export CARGO_BUILD_TARGET="x86_64-unknown-linux-gnu"
made it work.
Then @inferno-chromium it seems I am back to the previous problem of the fuzz target mixing C and Rust : I get
==11==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0xd71b43 in core::result::Result$LT$T$C$E$GT$::unwrap::h94893129de2c63ee /rustc/d8878868c8d7ef3779e7243953fc050cbb0e0565/src/libcore/result.rs:1004:13
#1 0xd71b43 in rs_dns_udp_register_parser /src/suricata/rust/src/dns/dns.rs:993:24
with the line being let default_port = std::ffi::CString::new("[53]").unwrap();
@catenacyber - i have fixed this now in https://github.com/google/oss-fuzz/pull/3859 , base build images should be rebuilt in one hour, then you need to infra/helper.py pull_images and retry.
Ok for MSan, check_build fails (which actually runs the binaries). @alexcrichton , can you please check below. Many projects use cargo fuzz build, is there a difference.
python infra/helper.py pull_images
python infra/helper.py build_fuzzers --sanitizer memory mp4parse-rust
python infra/helper.py check_build --sanitizer memory mp4parse-rust
Running: docker run --rm --privileged -i -e FUZZING_ENGINE=libfuzzer -e SANITIZER=memory -e ARCHITECTURE=x86_64 -v /build/oss-fuzz/build/out/mp4parse-rust:/out -t gcr.io/oss-fuzz-base/base-runner test_all
INFO: performing bad build checks for /tmp/not-out/avif.
INFO: performing bad build checks for /tmp/not-out/mp4.
Broken fuzz targets (2):
avif:
BAD BUILD: /tmp/not-out/avif seems to have either startup crash or exit:
INFO: Seed: 1337
INFO: Loaded 1 modules (25295 inline 8-bit counters): 25295 [0x55eb5fd167a8, 0x55eb5fd1ca77),
INFO: Loaded 1 PC tables (25295 PCs): 25295 [0x55eb5fd1ca78,0x55eb5fd7f768),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
==78==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x55eb5f4fa87f in rust_fuzzer_test_input /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20
#1 0x55eb5f71173f in __rust_try (/tmp/not-out/avif+0x36873f)
#2 0x55eb5f711324 in std::panicking::try::hfc14f47abd791d38 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panicking.rs:274:15
#3 0x55eb5f711324 in std::panic::catch_unwind::h3c37ac4d75f6cfa4 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panic.rs:394:14
#4 0x55eb5f711324 in LLVMFuzzerTestOneInput /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:25:22
#5 0x55eb5f786a4d in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:556:15
#6 0x55eb5f78af58 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:743:3
#7 0x55eb5f78c71e in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:794:3
#8 0x55eb5f726f28 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:826:6
#9 0x55eb5f70f950 in main /src/libfuzzer/FuzzerMain.cpp:19:10
#10 0x7fcf97fc382f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#11 0x55eb5f473398 in _start (/tmp/not-out/avif+0xca398)
DEDUP_TOKEN: rust_fuzzer_test_input--__rust_try--std::panicking::try::hfc14f47abd791d38
SUMMARY: MemorySanitizer: use-of-uninitialized-value /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20 in rust_fuzzer_test_input
ORIGIN: invalid (0). Might be a bug in MemorySanitizer origin tracking.
This could still be a bug in your code, too!
Unique heap origins: 35
Stack depot allocated bytes: 2376
Unique origin histories: 10
History depot allocated bytes: 240
Exiting
MS: 0 ; base unit: 0000000000000000000000000000000000000000
artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:
mp4:
BAD BUILD: /tmp/not-out/mp4 seems to have either startup crash or exit:
INFO: Seed: 1337
INFO: Loaded 1 modules (25344 inline 8-bit counters): 25344 [0x556b13d2f7d8, 0x556b13d35ad8),
INFO: Loaded 1 PC tables (25344 PCs): 25344 [0x556b13d35ad8,0x556b13d98ad8),
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
==82==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x556b135130dc in rust_fuzzer_test_input /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20
#1 0x556b13729f9f in __rust_try (/tmp/not-out/mp4+0x369f9f)
#2 0x556b13729b84 in std::panicking::try::hfc14f47abd791d38 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panicking.rs:274:15
#3 0x556b13729b84 in std::panic::catch_unwind::h3c37ac4d75f6cfa4 /rustc/0aa6751c19d3ba80df5b0b02c00bf44e13c97e80/src/libstd/panic.rs:394:14
#4 0x556b13729b84 in LLVMFuzzerTestOneInput /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:25:22
#5 0x556b1379f2ad in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) /src/libfuzzer/FuzzerLoop.cpp:556:15
#6 0x556b137a37b8 in fuzzer::Fuzzer::ReadAndExecuteSeedCorpora(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:743:3
#7 0x556b137a4f7e in fuzzer::Fuzzer::Loop(std::__1::vector<fuzzer::SizedFile, fuzzer::fuzzer_allocator<fuzzer::SizedFile> >&) /src/libfuzzer/FuzzerLoop.cpp:794:3
#8 0x556b1373f788 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) /src/libfuzzer/FuzzerDriver.cpp:826:6
#9 0x556b137281b0 in main /src/libfuzzer/FuzzerMain.cpp:19:10
#10 0x7f608ce0782f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x2082f)
#11 0x556b1348ac18 in _start (/tmp/not-out/mp4+0xcac18)
DEDUP_TOKEN: rust_fuzzer_test_input--__rust_try--std::panicking::try::hfc14f47abd791d38
SUMMARY: MemorySanitizer: use-of-uninitialized-value /rust/registry/src/github.com-1ecc6299db9ec823/libfuzzer-sys-0.3.2/src/lib.rs:124:20 in rust_fuzzer_test_input
ORIGIN: invalid (0). Might be a bug in MemorySanitizer origin tracking.
This could still be a bug in your code, too!
Unique heap origins: 35
Stack depot allocated bytes: 2392
Unique origin histories: 10
History depot allocated bytes: 240
Exiting
MS: 0 ; base unit: 0000000000000000000000000000000000000000
artifact_prefix='./'; Test unit written to ./crash-da39a3ee5e6b4b0d3255bfef95601890afd80709
Base64:
ERROR: 100% of fuzz targets seem to be broken. See the list above for a detailed information.
Check build failed.
Ah sorry I don't think I'm knowledgeable enough about fuzzers/msan/etc to understand what's going on there. Given "Might be a bug in MemorySanitizer origin tracking." it looks like something possibly wasn't compiled correctly? I don't have much experience myself in using other fuzzers in Rust. It might be worthwhile to raise this on the cargo-fuzz repo and see if others can help out?
@catenacyber @victorjulien, just fyi suricata project build seems broken due to recent default all Rust flags [https://github.com/google/oss-fuzz/pull/3859]. You would need to remove stable cargo from Dockerfile, use rust nightly directly(no need to install) and add this target CARGO_BUILD_TARGET="x86_64-unknown-linux-gnu" to make it work with sanitizer. I could not go far beyond that, there is some weird error - "make[2]: *** No rule to make target '../rust/target/release/libsuricata.a', needed by 'suricata'. Stop." python butler/infra.py pull_images python butler/infra.py build_images suricata
@inferno-chromium I will take a good look at it next week
The "weird error" is that Suricata build system is seeking the rust static library in some path which is not compatible with the use of CARGO_BUILD_TARGET
Quick and dirty patch I have been using is :
diff --git a/configure.ac b/configure.ac
index 127848234..747b96afb 100644
--- a/configure.ac
+++ b/configure.ac
@@ -2487,7 +2487,11 @@ fi
if test "x$cross_compiling" = "xyes"; then
RUST_SURICATA_LIB_XC_DIR="${host_alias}/"
else
- RUST_SURICATA_LIB_XC_DIR=
+ if test "x$CARGO_BUILD_TARGET" = "x"; then
+ RUST_SURICATA_LIB_XC_DIR=
+ else
+ RUST_SURICATA_LIB_XC_DIR="${CARGO_BUILD_TARGET}/"
+ fi
fi
if test "x$enable_debug" = "xyes"; then
I will try to adding export cross_compiling=yes
in build.sh
Quick and dirty patch seems good in fact https://github.com/OISF/suricata/pull/4986 Another PR to oss-fuzz is coming as well
Fyi, msan can't be enabled due to https://github.com/google/oss-fuzz/pull/3988#issuecomment-644230600
Is there a current work to have MSAN with recompiled std library ?
Is there a current work to have MSAN with recompiled std library ?
I dont know what is required here or changes needed in cargo fuzz.
For simple cargo projects the minimal set of flags required for MemorySanitizer would be:
$ export CC=clang CFLAGS=-fsanitize=memory RUSTFLAGS=-Zsanitizer=memory RUSTDOCFLAGS=-Zsanitizer=memory
$ cargo test --workspace -Z build-std --target x86_64-unknown-linux-gnu
-Z build-std
- rebuilds and instruments the Rust standard library.--target ...
- required for -Z build-std
. Additionally, no longer applies RUSTFLAGS
to procedural macros and build-scripts; exactly what we would like to do when using a sanitizer.CC
& CFLAGS
- the Rust standard library vendors libbacktrace which needs to be instrumented to avoid false positives with RUST_BACKTRACE=1
(in otherwise pure Rust code base).For projects using cargo-fuzz it would be additionally:
$ export CUSTOM_LIBFUZZER_PATH=...
$ cargo fuzz run fuzz_target -s memory -Z build-std --target x86_64-unknown-linux-gnu
CUSTOM_LIBFUZZER_PATH
- using prebuild libfuzzer library is probably preferable to the one vendored with cargo-fuzz. For example, with vendored one & CXXFLAGS
containing -fsanitize=memory
one would be instrumenting a fuzzer engine itself.Thanks @tmiasko it looks like it works cf https://github.com/OISF/suricata/pull/5395
rust-lang/rust#34701 is merged now which should help with getting coverage information.
FWIW, cargo fuzz
version 0.10.0 now has a cargo fuzz coverage
command that will build the fuzz target(s) with coverage instrumentation and then run the fuzz target(s) over the corpus and generate a merged coverage data file for the corpus.
Thanks for this news @fitzgen
Does it handle source remaps cf https://github.com/google/oss-fuzz/pull/5392
And removing optimization cf https://github.com/rust-lang/rust/issues/82144 ?
Could this be cargo fuzz -s coverage
as another sanitizer ?
Does it handle source remaps cf #5392
It does not. Is this something every user will need, or just oss-fuzz style users that move the binaries to new locations/machines? It hasn't been necessary for any of the local coverage experiments I've performed yet.
And removing optimization cf rust-lang/rust#82144 ?
Optimizations can be controlled via passing the -O
/--release
flag, and is orthogonal from coverage builds.
Agreed that the upstream rustc bug should get fixed, but I'm not convinced it warrants entangling optimized vs not builds together with coverage vs not builds.
Could this be
cargo fuzz -s coverage
as another sanitizer ?
Do you want the ability to build coverage binaries without automatically running them over the corpus to produce coverage data? We can definitely make it so that you can build a coverage-instrumented binary with cargo fuzz build --coverage
.
Does it handle source remaps cf #5392
It does not. Is this something every user will need, or just oss-fuzz style users that move the binaries to new locations/machines?
It is not about a new machine (this is handled by llvm-cov flags) This is rather about having absolute file paths cf https://github.com/rust-lang/cargo/issues/5450
It hasn't been necessary for any of the local coverage experiments I've performed yet.
You can try wasmtime from oss-fuzz You can try the regalloc repository, running it from the fuzz directory
And removing optimization cf rust-lang/rust#82144 ?
Optimizations can be controlled via passing the
-O
/--release
flag, and is orthogonal from coverage builds.Agreed that the upstream rustc bug should get fixed, but I'm not convinced it warrants entangling optimized vs not builds together with coverage vs not builds.
It may be better to wait for the rustc bug fix...
Could this be
cargo fuzz -s coverage
as another sanitizer ?Do you want the ability to build coverage binaries without automatically running them over the corpus to produce coverage data?
Yes :-)
We can definitely make it so that you can build a coverage-instrumented binary with
cargo fuzz build --coverage
.
For oss-fuzz projects, it could be easier with cargo fuzz build -s coverage
as we can use cargo fuzz build -s $SANITIZER
with SANITIZER being an environment variable that can be address, memory, coverage...
Tracking bug for ideas / suggestions