search

theRockLiu / googletest

Automatically exported from code.google.com/p/googletest
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

Death tests don't pass when using FAIL() in a child process on Windows. #234

Closed GoogleCodeExporter closed 9 years ago

GoogleCodeExporter commented 9 years ago 
Here's the sample code:

#include <gtest/gtest.h>
void foo() {
  FAIL() << "Test";
}
TEST(DeathTest, FailTest) {
  ASSERT_DEATH(foo(), "");
}

It works ok on Linux and Mac, but fails on Windows.
After a bit of debugging I think this is because "exit(1") is not called inside 
"UnitTest::AddTestPartResult" 
(gtest/src/gtest.cc).

This test passes if I set "::testing::FLAGS_gtest_throw_on_failure = true;"

I faced this issue in Chromium since its ~LogMessage with type=FATAL calls 
FAIL().
As a result, we can't write cross-platform death tests for Chromium without the 
hack described above.

Original issue reported on code.google.com by timurrrr@chromium.org on 7 Dec 2009 at 10:42

GoogleCodeExporter commented 9 years ago 
This seems to be a collision between the macros defined by Google Test and 
Chromium. Google Test defines its own macro called FAIL (see 
http://code.google.com/p/googletest/wiki/GoogleTestAdvancedGuide#Explicit_Succes
s_and_Failure). Google Test's FAIL only records a failure but doesn't terminate 
the 
program. In fact, this test program should always fail unless you specify flags 
such as 
--gtest_break_on_failure or --gtest_throw on_failure. Are you compiling it with 
Chromium headers included?

Original comment by vladlosev on 8 Dec 2009 at 3:57

GoogleCodeExporter commented 9 years ago 
The given program is compiled stand-alone, it uses FAIL from googletest.
I've tried it only on Windows.

http://codereview.chromium.org/462042/patch/1/3 was compiled this inside
Chromium and {DCheck,Check}DeathTest failed on Windows while passing on Linux 
and Mac.
Chromium has its own CHECK which does which calls a hook which uses 
googletest's FAIL
(names for these functions are given at 
http://code.google.com/p/chromium/issues/detail?id=24885 comment #6)

If this is a misuse of FAIL, I'd really appreciate if you can help me explain 
that to Chromium devs.

Original comment by timurrrr@chromium.org on 8 Dec 2009 at 5:32

GoogleCodeExporter commented 9 years ago 
For ASSERT_DEATH(statement, test) to pass, the program has to die somewhere 
inside statement. 
The control should NOT reach the end of statement, either in a natural way, or 
by using return. 
Looking at the Chromium source code, the default implementation of CHECK 
crashes the 
program on false condition. But if one sets a custom hook, this behavior may 
change and thus 
tests may fail. In order for death tests to pass, please make sure that 
whatever hook you are 
using is crashing the program on all platforms. Putting the FAIL() invocation 
into the hook makes 
no sense as any side effects (except for writing to stdout/stderr) produced by 
statement are 
discarded.

An alternative would be to not use death tests but just verify that your hook 
gets invoked when 
CHECK is triggered.

WRT your test program, I've just tried this code and it is generating failures 
on all three 
platforms, as expected:

*** On Windows:

H:\vladl\dev\vladl-test\snipped\gtest>cl test.cc src\gtest-all.cc 
src\gtest_main.cc -Iinclude -I. 
-EHsc
Microsoft (R) 32-bit C/C++ Optimizing Compiler Version 14.00.50727.762 for 80x86

Copyright (C) Microsoft Corporation.  All rights reserved.

test.cc
gtest-all.cc
gtest_main.cc
Generating Code...
Microsoft (R) Incremental Linker Version 8.00.50727.762
Copyright (C) Microsoft Corporation.  All rights reserved.

/out:test.exe
test.obj
gtest-all.obj
gtest_main.obj

H:\vladl\dev\vladl-test\snipped\gtest>test.exe
Running main() from gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DeathTest
[ RUN      ] DeathTest.FailTest
Running main() from gtest_main.cc
test.cc(6): error: Death test: foo()
    Result: failed to die.
 Error msg:
[  FAILED  ] DeathTest.FailTest (78 ms)
[----------] 1 test from DeathTest (78 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (78 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DeathTest.FailTest

 1 FAILED TEST

*** On Macintosh:

vladl-macbookpro:gtest vladl$ g++ --version
i686-apple-darwin9-g++-4.0.1 (GCC) 4.0.1 (Apple Inc. build 5493)
Copyright (C) 2005 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

vladl-macbookpro:gtest vladl$ g++ -Iinclude -I. -o test.out test.cc 
src/gtest-all.cc 
src/gtest_main.cc
vladl-macbookpro:gtest vladl$ ./test.out
Running main() from gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DeathTest
[ RUN      ] DeathTest.FailTest
test.cc:6: Failure
Death test: foo()
    Result: failed to die.
 Error msg: 
[  FAILED  ] DeathTest.FailTest (1 ms)
[----------] 1 test from DeathTest (2 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (2 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DeathTest.FailTest

 1 FAILED TEST

*** On Linux:

vladl@hungrymoose:~/dev/vladl-test/snipped/gtest$ g++ --version
g++ (GCC) 4.2.4 (Ubuntu 4.2.4-1ubuntu4)
Copyright (C) 2007 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

vladl@hungrymoose:~/dev/vladl-test/snipped/gtest$ g++ -Iinclude -I. -o test.out 
test.cc 
src/gtest-all.cc src/gtest_main.cc
vladl@hungrymoose:~/dev/vladl-test/snipped/gtest$ ./test.out
Running main() from gtest_main.cc
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from DeathTest
[ RUN      ] DeathTest.FailTest

[WARNING] ./src/gtest-death-test.cc:741:: Death tests use fork(), which is 
unsafe particularly in 
a threaded context. For this test, Google Test couldn't detect the number of 
threads.
test.cc:6: Failure
Death test: foo()
    Result: failed to die.
 Error msg: 
[  FAILED  ] DeathTest.FailTest (1 ms)
[----------] 1 test from DeathTest (1 ms total)

[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1 ms total)
[  PASSED  ] 0 tests.
[  FAILED  ] 1 test, listed below:
[  FAILED  ] DeathTest.FailTest

 1 FAILED TEST

Are you using a particular set of flags to compile it?

Original comment by vladlosev on 9 Dec 2009 at 10:21

GoogleCodeExporter commented 9 years ago 
Thanks, now I see the cause of the problem, it is Chromium-specific.

The hook I was talking about is only set on Windows, so that's a FAIL misuse:
http://src.chromium.org/viewvc/chrome/trunk/src/base/test/test_suite.h?revision=
33565
-> ctrl+f "SetLogAssertHandler"

Original comment by timurrrr@chromium.org on 9 Dec 2009 at 7:51

GoogleCodeExporter commented 9 years ago

Original comment by w...@google.com on 9 Dec 2009 at 11:05