search

google / benchmark

A microbenchmark support library
Apache License 2.0
9.03k stars 1.63k forks source link

[BUG] some tests segfault sporadically on OSX #1279

Open oontvoo opened 3 years ago

oontvoo commented 3 years ago

Describe the bug

System Which OS, compiler, and compiler version are you using:

To reproduce

  1. at HEAD
  2. cmake --build "build" --config Debug
  3. cmake -E chdir "build" ctest ---build-config Debug

Expected behavior All tests should pass at HEAD

Actual behavior The following tests FAILED: 47 - register_benchmark_test (SEGFAULT) 59 - user_counters_tabular_test (SEGFAULT)

Keep re-running the tests (step 3), one or both of them could pass occasionally. There seems to be some level of indeterminism here but I haven't been able to track down.

oontvoo commented 3 years ago

more logs:

The following tests FAILED:
     47 - register_benchmark_test (SEGFAULT)
     59 - user_counters_tabular_test (SEGFAULT)
Errors while running CTest
Output from these tests are in: /Users/vyng/repo/benchmark/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
vyng-macbookpro2 /Users/vyng/repo/benchmark$ clang --version                                                                      
Apple clang version 13.0.0 (clang-1300.0.29.3)
Target: x86_64-apple-darwin20.6.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
vyng-macbookpro2 /Users/vyng/repo/benchmark$ cmake -E chdir "build" ctest --rerun-failed --output-on-failure --build-config Debug 
Test project /Users/vyng/repo/benchmark/build
    Start 47: register_benchmark_test
1/2 Test #47: register_benchmark_test ..........   Passed    0.16 sec
    Start 59: user_counters_tabular_test
2/2 Test #59: user_counters_tabular_test .......***Exception: SegFault  0.01 sec
Running benchmarks...

50% tests passed, 1 tests failed out of 2

Total Test time (real) =   0.18 sec

The following tests FAILED:
     59 - user_counters_tabular_test (SEGFAULT)
Errors while running CTest
vyng-macbookpro2 /Users/vyng/repo/benchmark$ cmake -E chdir "build" ctest --rerun-failed --output-on-failure --build-config Debug 
Test project /Users/vyng/repo/benchmark/build
    Start 59: user_counters_tabular_test
1/1 Test #59: user_counters_tabular_test .......***Exception: SegFault  0.02 sec
Running benchmarks...

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.05 sec

The following tests FAILED:
     59 - user_counters_tabular_test (SEGFAULT)
Errors while running CTest
vyng-macbookpro2 /Users/vyng/repo/benchmark$ cmake -E chdir "build" ctest --rerun-failed --output-on-failure --build-config Debug 
Test project /Users/vyng/repo/benchmark/build
    Start 59: user_counters_tabular_test
1/1 Test #59: user_counters_tabular_test .......***Exception: SegFault  0.02 sec
Running benchmarks...

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.02 sec

The following tests FAILED:
     59 - user_counters_tabular_test (SEGFAULT)
Errors while running CTest
vyng-macbookpro2 /Users/vyng/repo/benchmark$ cmake -E chdir "build" ctest --rerun-failed --output-on-failure --build-config Debug 
Test project /Users/vyng/repo/benchmark/build
    Start 59: user_counters_tabular_test
1/1 Test #59: user_counters_tabular_test .......***Exception: SegFault  0.02 sec
Running benchmarks...

0% tests passed, 1 tests failed out of 1

Total Test time (real) =   0.02 sec

The following tests FAILED:
     59 - user_counters_tabular_test (SEGFAULT)
Errors while running CTest
vyng-macbookpro2 /Users/vyng/repo/benchmark$ cmake -E chdir "build" ctest --rerun-failed --output-on-failure --build-config Debug 
Test project /Users/vyng/repo/benchmark/build
    Start 59: user_counters_tabular_test
1/1 Test #59: user_counters_tabular_test .......   Passed    0.28 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.29 sec
dmah42 commented 2 years ago

seems like https://github.com/google/benchmark/blob/fcef4fb6697c78d7b7ace62da1bf7d6e329f5b1e/src/console_reporter.cc#L60 is not triggering correctly.

dmah42 commented 2 years ago

deleted other debugging comments because PEBKAC.

dmah42 commented 2 years ago

i can't reproduce this issue locally with apple clang 13.0. i can reproduce something similar by not passing the arguments correctly through to ctest though.

oontvoo commented 2 years ago

i can't reproduce this issue locally with apple clang 13.0. i can reproduce something similar by not passing the arguments correctly through to ctest though.

can you clarify what "not passing the args correctly" means? typos in the cmake command? Or did you mean there's a bug somewhere in cmake?

dmah42 commented 2 years ago

i mean i can create something that looks similar to the issue by running the test without passing the arguments through, but as far as i can tell ctest is doing the right thing (on my machine). trying to give you some threads to work with :)