Add `catkin test` verb

[timonegk]

This pull request adds the long-awaited catkin test verb.

Summary of the changes

The test verb aims to replace the current aliases for test and run_tests. Currently, calling catkin test does the following:

• For each of the requested packages, check if it is a catkin or a cmake package
• If it is a catkin package, execute make run_tests in the package
• If it is a plain cmake package, execute make test in the package
• Then, call catkin_test_results for the results The tests are executed in parallel, that can be prevented with -p 1. The test output is shown after all the tests of a package are completed, that can be changed with -i.

Things to do before the merge

• [x] add some tests
• [x] test on plain cmake packages
• [x] handle cmake packages that don't expose a test make target
• [x] make it possible to pass other test targets than run_tests and test
• [x] add shell completion
• [x] update documentation
• [x] fix --verbose
• [x] call make cmake_check_build_system before running tests
• [ ] make test output directory configurable
• [x] avoid filtering too much output on python packages
• [x] remove the manually incremented version
• [x] add --make-args

Test it

Please test the new verb on your setup and report any problems or requests in this thread! You can install this version with the command pip3 install git+https://github.com/catkin/catkin_tools.git@catkin_test. If the installation was successful, catkin --version should give the version 0.8.0.

[mikepurvis]

Nice work— seems reasonable to me. I tried some time ago but attempted that more ambitious path of trying to reach inside packages and then run individual test targets under catkin's parallelism:

https://github.com/mikepurvis/catkin_tools_test/blob/master/catkin_tools_test/test.py

Unsurprisingly, this was a total failure; your more restrained approach is the right way to go.

[ftsell]

I cannot attest to the complete correctness or code styles but I have tested this PR locally on some packages with different options and it works very well for me.

[Axel13fr]

Cheers @timonegk, I've used it on a couple of packages and that looked good to me. Tried with -i, -p options on catkin ROS packages with gtest and rostests.

[asherikov]

I think --make-args is necessary for this verb still, for example in order to pass ctest parameters which are accepted by cmake test target via ARGS: make test ARGS="--output-log a.log"

[timonegk]

Thanks for all of the positive feedback! I just added the --make-args option, maybe you could test if it works correctly @asherikov. I would like to open a new issue for the last point that is still unchecked (making the output directory configurable) to be done later and get this merged soon, as it is already a huge improvement over the existing state.

[asherikov]

Thanks,

• catkin test --test-target test --make-args "ARGS=--output-log $(pwd)/a.log --verbose" -- <pkg> -- works
• catkin test --test-target test --make-args "ARGS=--output-log $(pwd)/a.log --verbose" -j5 -- <pkg> -- does not, unrecognized option, not relevant in this case since the number of jobs can be set with ctest parameter, but may be there are other useful make parameters.

[LucasHaug]

Tested here and works very well!! Congrats

[timonegk]

@asherikov, I fixed your issue. I am going to merge this now.