Segmentation Fault when using `AddressSanitizer` with `address_space_limit_mb`

[DonggeLiu]

Segmentation Fault

Centipede runs into the following segmentation fault after a check failure:

centipede.cc:543] Batch execution failed; exit code: 1
Log of batch follows: [[[==================
Centipede fuzz target runner; argv[0]: /path/to/target/scarecrow.address flags: :timeout_in_seconds=1200::address_space_limit_mb=8192::rss_limit_mb=4096::crossover_level=50::shmem:arg1=/centipede-shm1-1824572-139817135887936:arg2=/centipede-shm2-1824572-139817135887936:failure_description_path=/tmp/centipede-1824572-139817135887936/failure_description:
timeout_in_seconds: 1200 rss_limit_mb: 4096
=================================================================
mmap failed: Cannot allocate memory
ERROR: Failed to mmap
==================]]]
centipede.cc:552] ReportCrash[0]: the crash occurred when running  /path/to/target/scarecrow.address on 1000 inputs
centipede.cc:594] ReportCrash[0]: executing inputs one-by-one, trying to find the reproducer
centipede_callbacks.cc:143] Check failed: !!(batch_result.Read(outputs_blobseq_))!=false [[1]    1824572 segmentation fault  $CENTIPEDE/bazel-bin/centipede --workdir= /path/to/target/workdir --corpus_dir= /path/to/target/corpus

Reproduce

The SegFault can be reproduced with Scarecrow, a minimum target with a planted memory leak that can be captured by the Address Sanitizer. The sanitized and unsanitized targets are respectively built by:

# Build target with ASAN
clang++ -fsanitize-coverage=trace-loads -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fno-builtin -fsanitize-coverage=trace-pc-guard,pc-table,trace-cmp -O2 -gline-tables-only  -ldl -lrt -lpthread -std=c++11 -Ilib/ -fsanitize=address scarecrow.cc  -o scarecrow.address  $CENTIPEDE/bazel-bin/libcentipede_runner.pic.a
# Build target without ASAN
clang++ -fsanitize-coverage=trace-loads -DFUZZING_BUILD_MODE_UNSAFE_FOR_PRODUCTION -fno-builtin -fsanitize-coverage=trace-pc-guard,pc-table,trace-cmp -O2 -gline-tables-only  -ldl -lrt -lpthread -std=c++11 -Ilib/ scarecrow.cc  -o scarecrow  $CENTIPEDE/bazel-bin/libcentipede_runner.pic.a

Then the SegFault can be reproduced by:

mkdir -p workdir corpus
$CENTIPEDE/bazel-bin/centipede --workdir=$PWD/workdir --corpus_dir=$PWD/corpus --fork_server=1 --exit_on_crash=0 --timeout=1200 --rss_limit_mb=4096 --address_space_limit_mb=8192 --binary=$PWD/scarecrow --extra_binaries=$PWD/scarecrow.address

While the following two commands are free from the SegFault and work fine:

# Disable address_space_limit
$CENTIPEDE/bazel-bin/centipede --workdir=$PWD/workdir --corpus_dir=$PWD/corpus --fork_server=1 --exit_on_crash=0 --timeout=1200 --rss_limit_mb=4096 --address_space_limit_mb=0 --binary=$PWD/scarecrow --extra_binaries=$PWD/scarecrow.address
# No sanitized binary
$CENTIPEDE/bazel-bin/centipede --workdir=$PWD/workdir --corpus_dir=$PWD/corpus --fork_server=1 --exit_on_crash=0 --timeout=1200 --rss_limit_mb=4096 --address_space_limit_mb=8192 --binary=$PWD/scarecrow

My Guess

Empty corpus file. The check failure and SegFault happened on the following line:

CHECK(batch_result.Read(outputs_blobseq_));

which points to the batch_result (which I reckon is the result?). File corpus.0 and directory corpus/ are both empty when the SegFault happens; They were not empty in the other cases above. Maybe the testcase was not successfully saved into corpus.0 and corpus/?

Please let me know if you could reproduce the SegFault or if more information is required, thanks!

[ussuri]

Thanks for the report. We'll investigate.

[kcc]

My guess, w/o actually checking yet... In runner.cc, inside SetLimits() we have if constexpr (kCanUseRlimitAs) { ... kCanUseRlimitAs is supposed to be false when building with ASAN, but it is computed via #ifndef ADDRESS_SANITIZER, which is not what ASAN sets by default (in google-internal environment it does, for historical reasons).

[DonggeLiu]

Thanks! I guess I can verify this theory later today by manually setting ADDRESS_SANITIZER to true and re-run : )

[kcc]

In the OSS version, we build the centipede runner with the default bazel flags and then link this run-time against the target, asan or not. I.e. the runner will be the same, it will have ADDRESS_SANITIZER=0.

So, I guess a better solution is to check for asan/tsan/msan at run-time as opposed to at-compile time. On it.

[kcc]

This one can be fixed relatively easy, by checking for ASAN/TSAN/MSAN in the runner at run-time, as opposed to at compile time. #203 sent

I just realized we have a similar problem with the interceptors, which should be present in the non-asan build, and should not be present in the ASAN build. See https://github.com/google/centipede/blob/main/runner_interceptors.cc The solution we use internally is to build the centipede runner with the correct sanitizer. Not sure it easily apples to the OSS world

[DonggeLiu]

Thanks : ) I will test the new code today and keep everyone updated!

[DonggeLiu]

Just tested the latest code with a (short) experiment: It appears that the bug has been fixed : ) I will close this issue and update Centipede in OSS-Fuzz. Thanks again!