Add first-class support for source-based coverage collection

[ChrisTimperley]

It would be great, for a variety of reasons, to have the ability to easily collect source-based coverage information using ROSWire (e.g., fuzzing, testing, repair, etc.). To that end, it is the goal of this issue to provide a simple but powerful API that allows users to collect coverage for a variety of languages using arbitrary coverage collection mechanisms (e.g., sancov, gcov, pycoverage).

Realising this goal consists of two tasks:

1. Firstly, we have to figure our robust ways of collecting coverage information for C++ and Python programs for ROS. We should come up with a set of executable steps for collecting coverage that we can manually perform inside a container for a ROS system.
2. Secondly, armed with the knowledge of how to collect coverage for ROS systems, we need to design a suitable API for ROSWire that allows users to easily collect coverage information without baking assumptions into existing code (e.g., by adding flags and conditionals).

Part I: How to collect coverage

clang++

On Ubuntu 18.04.4, the following must be installed: clang, clang-tools.

To add coverage instrumentation to a C++ program via catkin_make and clang++.

For newer versions of clang++:

$ catkin_make clean
$ catkin_make -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DCMAKE_CXX_FLAGS="-g -fprofile-instr-generate -fcoverage-mapping"

For clang++6:

$ catkin_make clean
$ catkin_make -DCMAKE_CXX_COMPILER=clang++ \
  -DCMAKE_BUILD_TYPE=RelWithDebInfo \
  -DCMAKE_CXX_FLAGS="-g -fsanitize=address -fsanitize-coverage=bb,trace-pc-guard"

Note that clang++ must be installed inside the image for this to work. We can add a which clang++ and clang++ --version check to ROSWire to raise an exception or produce a warning if this isn't the case.

We need to add the following environment variable to enable trace file generation:

ASAN_OPTIONS="coverage=1:coverage_dir=/tmp/cov"

Running the program will produce *.sancov trace files that are written to the /tmp/cov directory. To obtain source coverage information (as opposed to binary-level coverage) from these files, we need to symbolise the coverage report using sancov -symbolize:

$ sancov -symbolize [something.sancov] [binary-for-node]

Where we need to use catkin_find [package] [binary-name] to find the binary for a node:

$ catkin_find simulator follow_the_gap
/ros_ws/devel/lib/simulator/follow_the_gap

gcov

pycoverage

To collect coverage with pycoverage, we need to use the following launch prefix: pycoverage run -p. The -p option is used to avoid multiple processes clobbering the .coverage file to which coverage results are written. When running as ${USER}, the .coverage files are written to the ${HOME}/.ros directory. To combine all of the individual coverage files into a single .coverage file, we need to call coverage combine inside the ${HOME}/.ros directory. (Note that this will delete those original coverage files.) Finally, we can get a JSON-based coverage report using coverage json.

• [x] Parse and flatten the XML launch file
• [x] Identify all nodes in the launch file
• [x] Identify the Python nodes in the launch file (check the shebang!)
• [ ] Add launch-prefix to flattened XML launch file
• [x] Write flattened launch file to temporary file inside container
• [x] Launch using temporary file

Related #347

References

• https://clang.llvm.org/docs/SourceBasedCodeCoverage.html

Part II: Providing an API for collecting coverage as part of ROSWire

Constraints:

• In keeping with the design philosophy of ROSWire, it is highly desirable that any design be immutable (to reduce complexity/state space and make it easier to write concurrent apps around ROSWire).
• To that end, it makes sense that the "instrumentation" of a system should be specified/take place before the user is able to interact with that system (i.e., we should use some sort of builder to construct a System instance).