AddressSanitizer reports container overflow during benchmarking

jonstewart commented 6 months ago

Describe the bug When a source file has been compiled with the AddressSanitizer, it reports a container overflow error during benchmarking, with the callstack inside of Catch::Benchmark::Benchmark::run.

Expected behavior The benchmark should run without complaint from the AddressSanitizer.

Reproduction steps Steps to reproduce the bug.

Compile and run this code with the AddressSanitizer enabled:

#include <catch2/benchmark/catch_benchmark_all.hpp>
#include <catch2/catch_test_macros.hpp>

#include <cstring>

TEST_CASE("asanTest") {
  const char *s = "hello, there, what an odd world we live in.";

  BENCHMARK("memchr") {
    return std::memchr(s, 'v', 43);
  };
}

I get this error:

Filters: "asanTest"
Randomness seeded to: 168296164

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
benchmarks is a Catch2 v3.5.3 host application.
Run with -? for options

-------------------------------------------------------------------------------
asanTest
-------------------------------------------------------------------------------
test/benchmarks/test.cpp:
...............................................................................

benchmark name                       samples       iterations    est run time
                                     mean          low mean      high mean
                                     std dev       low std dev   high std dev
-------------------------------------------------------------------------------
memchr                                         100          3696      1.848 ms =================================================================
==49512==ERROR: AddressSanitizer: container-overflow on address 0x000108b03880 at pc 0x00010247707c bp 0x00016f135900 sp 0x00016f1350b0
READ of size 800 at 0x000108b03880 thread T0
    #0 0x102477078 in wrap_memcpy+0x3fc (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x1b078)
    #1 0x100ce96d0 in std::__1::pair<double const*, double*> std::__1::__copy_trivial_impl[abi:v160006]<double const, double>(double const*, double const*, double*)+0x48 (benchmarks:arm64+0x1000216d0)
    #2 0x100ce94a0 in std::__1::pair<double const*, double*> std::__1::__copy_trivial::operator()[abi:v160006]<double const, double, 0>(double const*, double const*, double*) const+0x28 (benchmarks:arm64+0x1000214a0)
    #3 0x100ce93b8 in std::__1::pair<double const*, double*> std::__1::__unwrap_and_dispatch[abi:v160006]<std::__1::__overload<std::__1::__copy_loop<std::__1::_ClassicAlgPolicy>, std::__1::__copy_trivial>, double const*, double const*, double*, 0>(double const*, double const*, double*)+0x54 (benchmarks:arm64+0x1000213b8)
    #4 0x100ce9344 in std::__1::pair<double const*, double*> std::__1::__dispatch_copy_or_move[abi:v160006]<std::__1::_ClassicAlgPolicy, std::__1::__copy_loop<std::__1::_ClassicAlgPolicy>, std::__1::__copy_trivial, double const*, double const*, double*>(double const*, double const*, double*)+0x24 (benchmarks:arm64+0x100021344)
    #5 0x100ce9300 in std::__1::pair<double const*, double*> std::__1::__copy[abi:v160006]<std::__1::_ClassicAlgPolicy, double const*, double const*, double*>(double const*, double const*, double*)+0x24 (benchmarks:arm64+0x100021300)
    #6 0x100ce92c0 in double* std::__1::copy[abi:v160006]<double const*, double*>(double const*, double const*, double*)+0x24 (benchmarks:arm64+0x1000212c0)
    #7 0x100ce928c in double* std::__1::__uninitialized_allocator_copy[abi:v160006]<std::__1::allocator<double>, double, double, (void*)0>(std::__1::allocator<double>&, double const*, double const*, double*)+0x28 (benchmarks:arm64+0x10002128c)
    #8 0x100ce9154 in void std::__1::vector<double, std::__1::allocator<double>>::__construct_at_end<double const*, 0>(double const*, double const*, unsigned long)+0x44 (benchmarks:arm64+0x100021154)
    #9 0x100ce8fa8 in std::__1::vector<double, std::__1::allocator<double>>::vector<double const*, 0>(double const*, double const*)+0xb4 (benchmarks:arm64+0x100020fa8)
    #10 0x100ce590c in std::__1::vector<double, std::__1::allocator<double>>::vector<double const*, 0>(double const*, double const*)+0x28 (benchmarks:arm64+0x10001d90c)
    #11 0x100ce5698 in Catch::Benchmark::Detail::classify_outliers(double const*, double const*)+0x28 (benchmarks:arm64+0x10001d698)
    #12 0x100ce1180 in Catch::Benchmark::Detail::analyse(Catch::IConfig const&, std::__1::chrono::duration<double, std::__1::ratio<1l, 1000000000l>>*, std::__1::chrono::duration<double, std::__1::ratio<1l, 1000000000l>>*)+0x1cc (benchmarks:arm64+0x100019180)
    #13 0x100cd8e18 in void Catch::Benchmark::Benchmark::run<std::__1::chrono::steady_clock>()+0x5c4 (benchmarks:arm64+0x100010e18)
    #14 0x100cce114 in CATCH2_INTERNAL_TEST_7()+0x294 (benchmarks:arm64+0x100006114)
    #15 0x100dcd40c in Catch::(anonymous namespace)::TestInvokerAsFunction::invoke() const+0x18 (benchmarks:arm64+0x10010540c)
    #16 0x100daaaec in Catch::TestCaseHandle::invoke() const+0x20 (benchmarks:arm64+0x1000e2aec)
    #17 0x100daa944 in Catch::RunContext::invokeActiveTestCase()+0x2c (benchmarks:arm64+0x1000e2944)
    #18 0x100da8cbc in Catch::RunContext::runCurrentTest(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)+0x35c (benchmarks:arm64+0x1000e0cbc)
    #19 0x100da85d4 in Catch::RunContext::runTest(Catch::TestCaseHandle const&)+0x1c4 (benchmarks:arm64+0x1000e05d4)
    #20 0x100d41f60 in Catch::(anonymous namespace)::TestGroup::execute()+0x9c (benchmarks:arm64+0x100079f60)
    #21 0x100d414a8 in Catch::Session::runInternal()+0x300 (benchmarks:arm64+0x1000794a8)
    #22 0x100d41144 in Catch::Session::run()+0x54 (benchmarks:arm64+0x100079144)
    #23 0x100ce0f04 in int Catch::Session::run<char>(int, char const* const*)+0x64 (benchmarks:arm64+0x100018f04)
    #24 0x100ce0e50 in main+0x40 (benchmarks:arm64+0x100018e50)
    #25 0x18d4c50dc  (<unknown module>)

0x000108b03880 is located 0 bytes inside of 800-byte region [0x000108b03880,0x000108b03ba0)
allocated by thread T0 here:
    #0 0x1024bd78c in wrap__Znwm+0x74 (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x6178c)
    #1 0x100cdc604 in std::__1::vector<double, std::__1::allocator<double>>::reserve(unsigned long)+0x130 (benchmarks:arm64+0x100014604)
    #2 0x100ce1028 in Catch::Benchmark::Detail::analyse(Catch::IConfig const&, std::__1::chrono::duration<double, std::__1::ratio<1l, 1000000000l>>*, std::__1::chrono::duration<double, std::__1::ratio<1l, 1000000000l>>*)+0x74 (benchmarks:arm64+0x100019028)
    #3 0x100cd8e18 in void Catch::Benchmark::Benchmark::run<std::__1::chrono::steady_clock>()+0x5c4 (benchmarks:arm64+0x100010e18)
    #4 0x100cce114 in CATCH2_INTERNAL_TEST_7()+0x294 (benchmarks:arm64+0x100006114)
    #5 0x100dcd40c in Catch::(anonymous namespace)::TestInvokerAsFunction::invoke() const+0x18 (benchmarks:arm64+0x10010540c)
    #6 0x100daaaec in Catch::TestCaseHandle::invoke() const+0x20 (benchmarks:arm64+0x1000e2aec)
    #7 0x100daa944 in Catch::RunContext::invokeActiveTestCase()+0x2c (benchmarks:arm64+0x1000e2944)
    #8 0x100da8cbc in Catch::RunContext::runCurrentTest(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&)+0x35c (benchmarks:arm64+0x1000e0cbc)
    #9 0x100da85d4 in Catch::RunContext::runTest(Catch::TestCaseHandle const&)+0x1c4 (benchmarks:arm64+0x1000e05d4)
    #10 0x100d41f60 in Catch::(anonymous namespace)::TestGroup::execute()+0x9c (benchmarks:arm64+0x100079f60)
    #11 0x100d414a8 in Catch::Session::runInternal()+0x300 (benchmarks:arm64+0x1000794a8)
    #12 0x100d41144 in Catch::Session::run()+0x54 (benchmarks:arm64+0x100079144)
    #13 0x100ce0f04 in int Catch::Session::run<char>(int, char const* const*)+0x64 (benchmarks:arm64+0x100018f04)
    #14 0x100ce0e50 in main+0x40 (benchmarks:arm64+0x100018e50)
    #15 0x18d4c50dc  (<unknown module>)

HINT: if you don't care about these errors you may set ASAN_OPTIONS=detect_container_overflow=0.
If you suspect a false positive see also: https://github.com/google/sanitizers/wiki/AddressSanitizerContainerOverflow.
SUMMARY: AddressSanitizer: container-overflow (libclang_rt.asan_osx_dynamic.dylib:arm64e+0x1b078) in wrap_memcpy+0x3fc
Shadow bytes around the buggy address:
  0x000108b03600: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000108b03680: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000108b03700: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
  0x000108b03780: fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa
  0x000108b03800: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x000108b03880:[fc]fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
  0x000108b03900: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
  0x000108b03980: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
  0x000108b03a00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
  0x000108b03a80: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
  0x000108b03b00: fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc fc
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==49512==ABORTING

test/benchmarks/test.cpp:: FAILED:
due to a fatal error condition:
  SIGABRT - Abort (abnormal termination) signal

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

Abort trap: 6

Platform information:

OS: macOS Sonoma 14.3
Compiler+version: Apple clang version 15.0.0 (clang-1500.1.0.2.5), Target: arm64-apple-darwin23.3.0
Catch version: v3.5.3

Additional context Add any other context about the problem here.

compilation

g++ -std=gnu++11 -std=c++17 -DHAVE_CONFIG_H -I.  -I./src -I./include -I/opt/local/include    -stdlib=libc++ -W -Wall -Wextra -Wnon-virtual-dtor -pedantic -pipe -O3  -fsanitize=address -fno-omit-frame-pointer  -MT test/benchmarks/benchmarks-test.o -MD -MP -MF test/benchmarks/.deps/benchmarks-test.Tpo -c -o test/benchmarks/benchmarks-test.o `test -f 'test/benchmarks/test.cpp' || echo './'`test/benchmarks/test.cpp

I had to compile Catch2 with c++17 support, using cmake -DCMAKE_CXX_STANDARD=17 -DCMAKE_INSTALL_PREFIX=/Users/me/build .. as another project I was building made use assertions concerning string_view.

I will confess that I have not tested the above code in a project on its own yet. I experienced the overflow in a different test, wrote the above as a simpler example, used the filter feature to run only the new test case, and got it to repro as well. I'll try to isolate more, but the executable isn't doing anything other than a small number of test cases, with nothing complicated (if at all?) happening at static time.

horenmar commented 6 months ago

Did you compile Catch2 library with ASAN (and container overflow? I am not sure if it is compiled in by default, or needs to be turned on explicitly)?

ContainerOverflow check requires cooperation from the container, which in turn means that all TUs touching std::vector have to have it enabled.

jonstewart commented 6 months ago

TIL! I'm surprised I haven't run into this in the past — but I'm working on a new system, so some things are different. I did not build Catch2 with asan. There are no other asan issues if I turn this detection off.

Given the intrusive nature of container-overflow, has there been consideration of bringing back header-only usage? Having to rebuild Catch2 with different compiler options to suit the situation seems unsustainable.

Regardless, though, thank you for the quick reply and for Catch2 generally! I wrote my own unit test library because GTest/Boost Test/CppUnit were all terrible, but Catch2 has the right ergonomics and I've been able to abandon my own library.

horenmar commented 6 months ago

You can drop the amalgamated .cpp file into your project if you want and it will be compiled as part of your main project build. You can find it in extras/. However, if you have any C++ dependencies, you have to handle propagating compiler flags to your dependencies either way, as there are various ways to break ABI compatibility between them.

catchorg / Catch2

AddressSanitizer reports container overflow during benchmarking #2835