search

areaDetector / ADCore

The home of the core components of the EPICS areaDetector software. It includes base classes for drivers and code for all of the standard plugins.
https://areadetector.github.io/areaDetector
Other
20 stars 67 forks source link

Setup a unittest framework #69

Closed ulrikpedersen closed 8 years ago

ulrikpedersen commented 9 years ago

It is now possible to instantiate plug-ins and drivers without requiring a full IOC build with dbd, database, channel access server, etc. See for example simDetectorNoIOC

This ability makes it possible to use external frameworks for unittesting as plug-ins and drivers can be autonomously instantiated, controlled and monitored without the need for user input or client-side scripts.

This ticket is intended to start work on an areaDetector unittest framework as a demonstration. The initial goals are fairly basic to simply demonstrate a test system:

ulrikpedersen commented 9 years ago

The basic requirements to a good test framework in my opinion:

I have investigated and used (on Linux only) these frameworks in the past and will start setting up an example:

ulrikpedersen commented 9 years ago

Created branch "unittest-framework" to work on this issue.

Tagging for notifications on this ticket: @ajgdls @EdWarrick

anjohnson commented 9 years ago

Note that libCom and the build system from EPICS Base have support for building and running test programs, and the growing number of self-tests in both Base itself and the V4 modules all use this framework. There is an epicsUnitTest.h header for C and C++ code which generates TAP output (see chapter 19 in the AppDevGuide), and the build system provides several build TEST* target variables and 2 build targets 'runtests' and 'tapfiles' that will run the test programs for you. Jenkins has a TAP plugin for displaying the output from the 'tapfiles' target. I hope you will carefully consider using this framework.

ulrikpedersen commented 9 years ago

Note that libCom and the build system from EPICS Base have support for building and running test programs, and the growing number of self-tests in both Base itself and the V4 modules all use this framework. There is an epicsUnitTest.h header for C and C++ code which generates TAP output (see chapter 19 in the AppDevGuide), and the build system provides several build TEST* target variables and 2 build targets 'runtests' and 'tapfiles' that will run the test programs for you. Jenkins has a TAP plugin for displaying the output from the 'tapfiles' target. I hope you will carefully consider using this framework.

You're right of course, there is a unittest framework in EPICS base. For the V4 libraries which may (hopefully soon) merge into EPICS base, this is a natural and appropriate choice. However, for external modules/libraries like areaDetector I think a wider range of options can be considered.

On the unittest-framework branch we have been adding a setup based on the boost unittest framework. The boost libraries are the closest thing to an extension of the C++ standard and as such incredibly popular and very well tested by C++ developers. My "research" (non-scientific googleing and stackoverflow) found that the boost and the google test frameworks appear to compete for the top-spot in C++ testing. I am very much in favour of using generic and as-close-to-a-standard-as-possible tools like that.

The boost test features I find are suitable for working in areaDetector, in particular:

Google also has a "mock" framework (gmock is an addition to the gtest framework) which could be quite useful, but I have not yet investigated much further.

anjohnson commented 9 years ago

None of your other test frameworks are going to be able to run tests on VxWorks. For the camera plugins that obviously won't be a problem at all, but I believe that some of Mark's other modules for VxWorks I/O devices do use ADCore, so he might want to be able to test that code on VxWorks targets. Now you do have to do a bit more work to be able to run the tests on VxWorks with the EPICS framework, but it's not hard to do.

The easier it is for developers to build and run the tests, the more often they will do so, thus if you do pick an external framework I would still encourage you to keep the external module requirements to a minimum.

ulrikpedersen commented 9 years ago

That's true, we don't have boost on VxWorks, nor do we use STL - much to my disappointment. For areaDetector which supports only a very small subset of detectors on VxWorks, I am of the opinion that we should not let the lowest common denominator (VxWorks) dictate all future developments.

In fact at this time the VxWorks build of ADCore is limited to a subset of modules - for example no HDF5 support is built. Even with a boost unittest framework, ADCore will still build and run on VxWorks as well as it does now - and if anyone fancy adding VxWorks tests then by all means do so.

Boost is full of modern C++ utilities and I think it is a shame not to consider ways to make use of this in any modern C++ development. However, in the interest of practical use in areaDetector the external dependency on boost (unittest library only) is managed carefully - it is not enabled by default so unsuspecting users do not need to install boost. It is merely an option for developers to enable it for building and running tests.

anjohnson commented 9 years ago

FYI the V4 pvCommonCPP module contains a copy of Boost 1.53, although it only installs the subset of the headers needed for the V4 code (smart_ptr and its dependencies). This works on VxWorks 6.x which V4 supports; we dropped support for VxWorks 5.x in the latest V4 release.

I have also created a couple of EPICS modules (now available on the EPICS Modules Download page) containing boost 1.57.0 split into two parts, the smaller of which could be used instead of the pvCommonCPP module when building V4. This module (boost-1.57.0) is under 80MB of source code and provides more Boost features than pvCommonCPP currently does, while the other module (boost-extra-1.57.0) is 433MB and contains the remaining headers plus the Boost doc, libs and tools directories. I hope to replace pvCommonCPP with the boost-1.57.0 module before integrating the V4 modules into Base. It would really only be needed when building for targets that do not already provide Boost, but the main idea is to make installation easier for users.

If the ADCore tests were to use the Base test framework then they would not have to be rewritten from scratch to run on VxWorks (even if you don't try to actually do that yourselves), plus anyone who builds ADCore can run them (just type 'make runtests') to check for build problems without having to install any additional dependencies.

ulrikpedersen commented 9 years ago

It is interesting that you're creating epics boost modules. With those (probably boost-extra) you can then presumably use the boost unittest framework? ;-)

I find that boost is not really difficult to install at all from a package manager - and even building it from source is fairly easy (although not as trivial as configure;make;make install - and the actual build takes forever). Users of ADCore who are interested in running unittests are typically not end-users, but developers and are capable of installing boost. Also in order to properly build ADCore you really do need to install some external dependencies anyway!

My motivation for working with unittests is to help me with the daily development cycle (buzzword alert: TDD) and I expect other AD developers to have a similar wish. To make that work smoothly I require the ability to run a subset of the entire test framework, and seamless integration with my IDE. I use mainly Eclipse CDT which comes with support for google and boost test frameworks to do one-click run and parsing of results. With boost being such a standard utility other IDEs probably support a similar integration.

With regards to VxWorks support, I feel that if we have unittests that run and test on Linux (and maybe Windows) we cover the majority of areaDetector use-cases. If ADCore is thoroughly tested (we're far from there) on those host platforms then we will have more confidence that the code-base will work on VxWorks too.

anjohnson commented 9 years ago

Actually boost/test is in my main boost module — it wasn't huge and didn't need a separate library to be built, those being the main characteristics which caused me to push modules into the boost-extra module.

Installing boost from a package manager is fine for self-hosting target architectures, but it does nothing to help most cross-build architectures. Even where the OS is the same (such as when cross-building for linux-arm) the compiler cannot safely search the include directories of the host OS, so you still have to install a separate copy. That's multiple copies if you have to support multiple cross-architectures — the APS accelerator support modules are cross-built for 3 different VxWorks versions, 2 RTEMS targets and 3 different linux ARM targets so we would need 8 copies if each compiler had to have its own installation!

Base has a lot of tests written in C code, and the build system knows how to run our tests automatically. Matej had started developing tests for pvAccess using the gtest framework, but converted them to the EPICS Unit Test framework when I showed it to him. It's really not that bad, and I'd be happy to add new features. You can run each test program manually to see the pass/fail results for each check, after which the test framework then gives you a summary of the results, or you can use 'make runtests' to run all the tests either in a single directory or in the whole tree. Try it: cd into your base/src/libCom/test/O.linux-x86_64 directory and run ./epicsMathTest for example.

ulrikpedersen commented 9 years ago

We can keep going around this in circles, I think. The bottom line is that our requirements are somewhat different because we support different frameworks. I understand for EPICS base it is sensible to have unittests that works on a range of cross-compiled platforms - and make runtest is useful for a site-integrator to do a basic sanity check after installation into a new environment. I have tried running the tests in EPICS base - and it works. However, the areaDetector unittest users that I have in mind are the other developers in the collaboration - and cross-compilation is not a big part of areaDetector. We have only one (or maybe 2?) detectors supported for VxWorks - and then only supported with a subset of features as not all of ADCore builds on VxWorks. All other detectors are built and run on the host platform (which are supported by Boost).

I have had a very good experience with Boost test and it supports exactly my workflow where 'make runtests' is not necessary - but IDE integration is. I really don't want to convert my test system to a much smaller one with less features, fewer users, maintainers, examples, and docs - simply so that it can support a 'make runtests' target which I don't particularly need - and a VxWorks platform that I don't run tests on anyway as I don't have any detectors for it. That just doesn't make sense to me.

Of my original list of test frameworks: google test, cppunit, and Boost all works with my use-case - but I think Boost wins clearly due to the major use by C++ developers everywhere which in practice makes it a standard utility. As I mentioned before I'm all in favour of using standard tools and not re-inventing them.

ulrikpedersen commented 8 years ago

Unittest framework is now on master. Closing ticket.