Closed gualandrid closed 6 years ago
csoltenborn
commented
6 years ago Thanks for the comprehensive bug report! Could you please attach some code containing your problematic tests? Test methods can be empty... I'm rather busy with other things at the moment - this will help me track down and fix the issue...
gualandrid
commented
6 years ago Sure, thank you for investigating. Here is a stripped down version of the test code. All of the TEST_F() tests are discovered without issue, and the whole thing works fine when INSTANTIATE_TEST_CASE_P() and TEST_P() portions are commented out. When INSTANTIATE_TEST_CASE_P() and TEST_P() are included, the error occurs.
I also confirmed that running tests.exe --gtest_list_tests from the command line discovers all of the parameterized tests.
class Test_ReynoldsNumber : public ::testing::Test
{
protected:
static ExpectedOutputs re_data;
static void SetUpTestCase()
{
//load shared data from file into re_data
}
virtual void SetUp(){}
virtual void TearDown(){}
static void TearDownTestCase(){}
};
ExpectedOutputs Test_ReynoldsNumber::re_data;
TEST_F(Test_ReynoldsNumber, RhoVL_results)
{
// test using above test fixture to test a function
}
TEST_F(Test_ReynoldsNumber, PhiL_results)
{
// another test using above fixture
}
TEST_F(Test_ReynoldsNumber, gen)
{
// test of the parameter generator function
params = ParamGen();
}
class ReynoldsTestParam : public ::testing::TestWithParam<TestData<double, double>>
{
protected:
ReynoldsTestParam(){}
//static void SetUpTestCase(){}
//virtual void SetUp() {}
//virtual void TearDown() {}
//static void TearDownTestCase() {}
};
template<typename T1, typename T2>
struct TestData
{
std::vector<T1> input;
std::vector<T2> output;
};
std::vector<TestData<double, double>> ParamGen()
// instantiate parameter test case using ParamGen() function, which returns a std::vector of custom struct TestData<double, double>
INSTANTIATE_TEST_CASE_P(firstTest, ReynoldsTestParam, ::testing::ValuesIn(ParamGen()));
TEST_P(ReynoldsTestParam, param_rhoVL)
{
// use loaded parameters to test function
testData = GetParam();
EXPECT_DOUBLE_EQ(function_under_test(),expected_value);
}I guess you have stripped a bit too much code ;-) - the above doesn't compile for me... would you mind to fix that? After adding the gtest include, VS still reports 22 errors...
gualandrid
commented
6 years ago While working on a better example, I think I've narrowed it down somewhat. The ParamGen() function loads data from a file. When I modified ParamGen() to directly create some dummy parameters, tests were discovered correctly without error. When I have them load from file, the error occurs.
I've tried using an absolute path, and moving the data file to the build or project directories, but that didn't help.
gualandrid
commented
6 years ago Correction to last comment: an absolute path works, but relative paths did not. Do you know what the base of the relative path would be when the test adapter loads the test?
csoltenborn
commented
6 years ago I see... You can influence the working directory with the respective option. However, there's a bug which prevents test discovery from using that setting (fix will be released soon).
But I have a different recommendation: Make sure that your main method does not load test data etc. - use the according gtest constructs instead. The main method should basically look like here. See last comments of #196 which is probably kind of comparable to your case. Let me know whether this helps...
gualandrid
commented
6 years ago This is my whole main() function, it does not have any extra parts or loading to it, it is bare bones based on examples I could find. Is this what you are referring to (I'm still very new to using Google Test, and to C++ in general)?
The test data is loaded in using a static method in the test fixture classes (which are ParamInterface fixtures). It appears that Google Test usually runs the load method during the instantiation INSTANTIATE_TEST_CASE_P(), before the rest of the test setup is done.
If this sounds incorrect, I am open to suggestions! Thanks for your help so far.
#include "gtest/gtest.h"
int main(int argc, char **argv)
{
/*The method is initializes the Google framework and must be called before RUN_ALL_TESTS */
::testing::InitGoogleTest(&argc, argv);
/*RUN_ALL_TESTS automatically detects and runs all the tests defined using the TEST macro.
It's must be called only once in the code because multiple calls lead to conflicts and,
therefore, are not supported.
*/
return RUN_ALL_TESTS();
}Yes, that's what I meant, and your main method looks fine. My guess is that your static method is already executed by means of instantiating your test class (or something like this, as happened in #196).
I would put a breakpoint into your static method, configure the test project as the startup project, add the --gtest_list_tests option as an argument, and then debug your executable. You will then probably see that the method is executed during test discovery. If you then make sure that this execution does only happen at appropriate places (e.g. global setup), but not during test discovery, you will probably be fine.
gualandrid
commented
6 years ago I will take a look into that. The discovery is working for me as-is when using absolute paths, and the working directory fix may take care of it for relative paths, so I should be able to move forward.
I am a bit confused on how I would move the method execution to later in the process, since I am using the value-parameterized test fixtures. Is there a way to delay the loading of these parameters?
csoltenborn
commented
6 years ago Well, as I said, use the appropriate setup/teradown mechanisms. My understanding is that gtest will make sure that everything located at an "official setup/teardown place" will not be executed during test discovery. For instance, if I add constructor, destructor, SetUp(), and TearDown() methods to class ParameterizedTests of the SampleTests solution's Test project (that class is a parameterized test fixture!) and place breakpoints at all of them, no breakpoint is hit when running the project with --gtest_list_tests parameter, but they are all hit (several times) when running without that parameter. Thus, gtest provides this for free if you use it appropriately.
That's why I suggested to use breakpoints to actually figure out who is calling your code (and when).
I will now close this issue - feel free to re-open if you have more questions (or create a new issue)...
csoltenborn
commented
6 years ago One more hint: My guess is that your static variable re_data is causing the trouble. If I add such a variable and place a breakpoint at the variable's type's constructor, it is hit during test discovery. Stacktrace from test discovery (note that gtest is not involved here - this is basic C++ functionality, so to speak):
This is easily solved by making that variable a pointer and initializing/destroying the pointer at the appropriate setup/teardown place (see below for global setup/teardown). Stacktrace from test execution (breakpoint is not hit during discovery):
And that's pretty much exactly what happened with #196 ;-)
csoltenborn
commented
6 years ago P.S. Please update the issue with your actual cause...
gualandrid
commented
6 years ago Updated. Thanks for the quick response!
csoltenborn
commented
6 years ago You are welcome! However, I wanted to ask you to add a comment explaining the final issue with test discovery and how you solved it (not the issue's title, although that change is just fine)...
gualandrid
commented
6 years ago The root cause of my issue was that my parameterized tests were loading data from the file during test discovery (test initialization, where INSTANTIATE_TEST_CASE_P() is executed). The filepath was a relative path. Google Test Adapter would show an error using this configuration, but worked fine if the filepath is changed to an absolute path. When running the tests.exe --gtest_list_tests, which the GTA uses to discover tests, the GTA was not finding the files to load since it was not using the project working directory. Running tests.exe from the command line worked as usual since the relative file paths were valid.
This may be remedied by simply using absolute file paths, or moving the test data loading into a global or test case setup() function (which do not run during test discovery), instead of during test initialization. In the future, the relative paths should work once the working directory GTA bug is fixed (as mentioned by @csoltenborn above).
csoltenborn
commented
6 years ago Thanks!
I am encountering an error using the Test Adapter with value-parameterized tests. The full test suite runs normally when invoking the built test .exe from the command line, but I get the following output in Visual Studio after a build:
When run from the command line, it runs and exits fine, with this output (failed test is expected):
I found some other issues that were similar, but the solutions found there have not fixed my situation. The adapter works normally when the parameterized portions of the tests are commented out, so it seems to be driven by that portion of the test code.
My environment info: