C++ unit tests generate xUnit style reports

[rhyolight]

Depends on #9.

Current C++ unit tests only report to stdout and pass/fail via exit status. These tests need to generate xUnit style reports that can be published publicly. Here is a list of C++ unit testing frameworks.

[subutai]

Information on current test format and setup here: https://github.com/numenta/nupic/wiki/CPP-Unit-Tests

Currently uses a home grown testing framework. Would be great to transition all these to a more standard test framework.

[utensil]

My team at work have been using Google Test for a few years. I don't recall the process of choosing from frameworks and all the pros and cons, but it seems to be widely adopted and I do recall it's definitely much better than CppUnit(a direct naming correspondence to JUnit and xUnits) when I experimented with both of them.

Plus it gets better working with Google Mock.

[kandeel]

I felt very comfortable using xunit/amop because you can just define the mock inside the test you don't have to create a compile time mock class. But also gtest/gmock is great specially testing native code for different processor architecture. we end up using both.

[rhyolight]

Thanks for your comments, guys. @fergalbyrne has volunteered to work on this within the next week, and @moorejpdx has has also told me he's willing to advise and help out as needed.

[fergalbyrne]

Quick update: I'm incorporating Google Tests into the CMake workflow, should have a sample set of tests in the next few days.

[rhyolight]

+1

Sent from my MegaPhone

On May 24, 2014, at 4:44 AM, Fergal Byrne notifications@github.com wrote:

Quick update: I'm incorporating Google Tests into the CMake workflow, should have a sample set of tests in the next few days.

— Reply to this email directly or view it on GitHub.

[rhyolight]

@fergalbyrne Any progress on this?

[fergalbyrne]

Hi Matt. Yes, I've added the correct elements to the cmake scripts, included the header files and the libs, etc. I've to add an actual test using a bit of the NuPIC codebase and then we'll be done. Please carry over into the next sprint, I'll have it closed in the next week, ready for people to start writing or transferring unit tests from Numenta's system to Google Test.

On Fri, Jun 6, 2014 at 8:18 PM, Matthew Taylor notifications@github.com wrote:

@fergalbyrne https://github.com/fergalbyrne Any progress on this?

— Reply to this email directly or view it on GitHub https://github.com/numenta/nupic.core/issues/10#issuecomment-45374277.

Fergal Byrne, Brenter IT

Author, Real Machine Intelligence with Clortex and NuPIC https://leanpub.com/realsmartmachines

Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014: http://euroclojure.com/2014/ and at LambdaJam Chicago, July 2014: http://www.lambdajam.com

http://inbits.com - Better Living through Thoughtful Technology http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne

e:fergalbyrnedublin@gmail.com t:+353 83 4214179 Join the quest for Machine Intelligence at http://numenta.org Formerly of Adnet editor@adnet.ie http://www.adnet.ie

[rhyolight]

@fergalbyrne I'm going to steal this issue from you, as it's becoming more important to close it out as I've added test tooling in NuPIC. The NuPIC Core test tooling is lacking. If you have any progress to share, please push to a remote branch and I'll take over.

[utensil]

Yes, if there's a public branch, I'm also willing to help.

[rhyolight]

@utensil I've merged in master and the my public branch is here: https://github.com/rhyolight/nupic.core/tree/google-test. I'm getting the following error on make:

[ 62%] Built target nupic_core
Scanning dependencies of target googletests
Scanning dependencies of target testcpphtm
[ 63%] Linking CXX executable /Users/mtaylor/nta/nupic.core/build/release/bin/googletests
Building CXX object CMakeFiles/testcpphtm.dir/test/htmtest/Htmtest.cpp.o
Undefined symbols for architecture x86_64:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

[utensil]

It looks like that a proper main entry is not given which is required by any C++ program. And this is most likely due to gtest is not set up correctly somehow, because gtest, as a test runner, should provide the main entry. I'll look into it.

[utensil]

I hava a different error:

make[2]: *** No rule to make target `/home/utensilsong/projects/nupic.core/external/linux64/lib/libgtest.a', needed by `/home/utensilsong/projects/nupic.core/build/release/bin/googletests'.  Stop.
make[1]: *** [CMakeFiles/googletests.dir/all] Error 2
make: *** [all] Error 2

This is because only mac version of google test is included in external.

[utensil]

@rhyolight The reason you encountered the problem is because for now, only the mac version of external/darwin64/lib/libgtest.a is included, but not libgtest_main.a and it's not linked.

[utensil]

@rhyolight The problem is fixed for linux64 on https://github.com/utensil/nupic.core/tree/10-google-test .

To fix it for darwin64, just:

svn checkout http://googletest.googlecode.com/svn/trunk/ googletest
cd googletest
cmake .
make
cp libgtest*.a $NUPIC_CORE/external/darwin64/lib/

[utensil]

The only issue is that I don't know which version of google test is @fergalbyrne using and the command above should be modified to point to a version tag of googletest instead of trunk.

[rhyolight]

@utensil There is already a https://github.com/utensil/nupic.core/blob/10-google-test/external/darwin64/lib/libgtest.a in the branch. Do I override it with the binary I created locally?

[utensil]

@rhyolight We should compile libgtest.a and libgtest_main.a based on gtest 1.7 for all OSes:

• darwin64 (you can)
• linux64 && linux32 (I can)
• Windows (I can)
• Arm v5 and v7 (I don't know who can)

and update google test header accordingly. It should be done consistently, so overwriting the previous provided .a is OK, and it should be done.

[utensil]

@olavks points out providing compiled binary library is "broken by design" here and partially agree with him. During working on this fix, I'm feeling the downside of this approach:

• hard to provide binaries for all OSes
• hard to provide them correctly

@rhyolight What do you think?

[rhyolight]

Is there any way to have cmake install the gtest dependency in a platform-specific way?

[utensil]

No, except include the source and compile it by CMake, I guess.

[rhyolight]

Because gtest is purely a test dependency, the only reason it needs to be installed is for developers and the build system. We need to ensure that gtest binaries are not included in the release folder, but otherwise, I don't have a problem manually compiling them and injecting them into the source.

[utensil]

New Goal

• [ ] Rework #150 using Google test. Will come up with a new PR ASAP.
• [x] Wiki is updated to reflect this change.

Why

@rhyolight mailed me(here is just a digest):

After talking to the guys here, we would really like you to continue your work, but we're concerned about using CATCH instead of a more time-tested and standard framework like googletest. We'd like to know what you think about it. Would you be willing to revert back to googletest? I couldn't find a way to get it working without committing the binaries into the source code, but maybe there's a better answer I'm simply missing.

And I replied:

It's possible to revert to googletest and I do have noticed one way to using google test without the need to include the binaries: just make gtest to generate gtest/gtest.h and gtest/gtest-all.cc as guided here: https://code.google.com/p/googletest/wiki/V1_7_AdvancedGuide#Fusing_Google_Test_Source_Files .

I chose Catch mainly for it's easier adoption and I was interested in experimenting something new and also promising. I understand the concern about not using a more time-tested and standard framework. If you guys are all in favor, I can rework them using google tests.

And @rhyolight replied(here is just a digest):

I think that's what Scott and Chetan would prefer.

[utensil]

Sorry for the long delay, the work mentioned above has started at https://github.com/utensil/nupic.core/compare/10-google-test-rewrite .

[utensil]

Finally, the hard part has been completed! I've successfully passed all of the test cases in src/test/testeverything/engine but one(caused by gtest compare char* strings by a different macro) under gtest.

See https://travis-ci.org/utensil/nupic.core/builds/34636169 . And for the re-implemented Tester(which prevented rewriting existing test cases in testeverything), see https://github.com/utensil/nupic.core/blob/10-google-test-rewrite/src/test/googletests/Tester.hpp .

This progress means we can indeed port existing test cases in testeverything to google tests with minimal modifications. The work left to be done is to integrate some goodies in #150 related to presenting the test result as an html and wrap it up.

[utensil]

@rhyolight This needs to be closed because of the merge of #179 and numenta/nupic#1409 .

All remaining work will be tracked at #183 .