Decide on a unit test framework

evanmayer commented 2 years ago

Task

Pick a unit test framework for the project and describe why it was chosen.

Acceptance

Comment on this issue to document process of choosing.

Provide a sample unit test project.

Details

evanmayer commented 2 years ago

Requirements

Easy to read - so new users can read existing tests and extend them easily
Mocking framework easy to integrate - many low level modules interface with hardware, so they must be mocked out to get test coverage
No external dependencies - should be small and easy to install, since it will likely need to be added to vagrant images and/or flight computers, and keeping external dependencies limited will help longevity
Good documentation - so I can avoid writing about how to write tests

Options Considered

Google Test

Pros:

Popular
Documented
Has gMock mocking framework

Cons:

Relatively large install
Need to write C++ code to test C
Mocking framework in gmock designed for C++, not C, making test code more difficult to write

Boost.Test

Pros:

Easy to write (at least for C++, in my experience)
Good documentation
Multiple options for including tests: header-only, static, dynamic library
Can do a standalone Boost.Test install: libboost-test-dev

Cons:

Header-only install is not a single header - the one header #includes many others (think about it - a single header would be a nightmare to maintain)
Would need separate mocking framework
Need to write C++ code to test C

Unity

Pros:

Easy to write
Excellent documentation
Good example code
Minimal install - 2 .h, 1, .c for unit tests
Only need to write C code to test C

Cons:

CMock mocking framework is separate, and has Ruby dependency
Mocking and test running solution pushes Ceedling ecosystem, which adds more dependencies

Check

Pros:

Simple interface
Easy to read documentation

Cons:

No mocking framework

Cpputest

Pros:

Well documented
Has mocking framework
Has books written about/with it
Package manager install is easy

Cons:

Large install with a lot of fluff (e.g. we don't need .vcxproj files for build)
Package manager install (packages may not be available for some OSs)

cmocka

Pros:

Small: source is 1 .c, 5 .h
Single dependency provides both unit test and mocking framework
Written for C in STD C
Has test fixtures

Cons:

Online docs/examples not as good as others - but more docs exist in repo
Only a few large projects (samba, libssh, OpenVPN) use it

Conclusion

Based on this, I recommend cmocka. Cpputest would also probably work.

cmocka example code in the SW primer repo.

Building tests against it was simple, and would be even easier to use if we used the included CMake files to install it in the dev environment, rather than building against it as a static library as I did in the example. This is the preferred method of using it; right now the cmocka unit tests are mixed in with mine because of the lazy CMake I wrote.

It's even possible to build it in the Debian environment by relaxing the CMake version requirement to 3.0.0.

CTest output:

vagrant@debian-jessie:/git/TIM-software-primer/unittest_example_cmocka/build$ ctest
Test project /git/TIM-software-primer/unittest_example_cmocka/build
    Start 1: add_example_test
1/7 Test #1: add_example_test .................   Passed    0.00 sec
    Start 2: simple_test
2/7 Test #2: simple_test ......................   Passed    0.00 sec
    Start 3: allocate_module_test
3/7 Test #3: allocate_module_test .............   Passed    0.00 sec
    Start 4: assert_macro_test
4/7 Test #4: assert_macro_test ................   Passed    0.00 sec
    Start 5: assert_module_test
5/7 Test #5: assert_module_test ...............   Passed    0.00 sec
    Start 6: waiter_test_wrap
6/7 Test #6: waiter_test_wrap .................   Passed    0.00 sec
    Start 7: test_uptime
7/7 Test #7: test_uptime ......................   Passed    0.00 sec

100% tests passed, 0 tests failed out of 7

Total Test time (real) =   0.04 sec

Running the test executable manually:

vagrant@debian-jessie:/git/TIM-software-primer/unittest_example_cmocka/build/src$ ./add_example_test_exec 
[==========] Running 1 test(s).
[ RUN      ] test_AddOnePlusOne
[       OK ] test_AddOnePlusOne
[==========] 1 test(s) run.
[  PASSED  ] 1 test(s).

evanmayer commented 2 years ago

However, we could also go the route of using a C unit testing framework that does only unit tests (Unity, Check), and tack on a mocking framework like FFF if/when we need it.

evanmayer commented 2 years ago

Nick has rewritten any tests he wrote in Unity using cmocka (thanks Nick!) so we'll go forward with that.

tim-balloon / TIMflight