search

bit-team / backintime

Back In Time - An easy-to-use backup tool for GNU Linux using rsync in the back
https://backintime.readthedocs.io
GNU General Public License v2.0
1.92k stars 178 forks source link

Test & Build: Add target "unittest" #1489

Open buhtz opened 11 months ago

buhtz commented 11 months ago

Depends on #1575 which need to be fixed first.

Problem

BITs test suite do have three kinds of tests in it: unit, integration and system tests. Integration and system tests are valuable but they do not run under every circumstances (e.g. chroot build environments or other sandboxes). Debian do has the problem (Debian#940319) for example.

Proposal

Separate between unittests and all other tests to account for the different characteristics of these test categories.

Solution

Currently BIT do use make as build system also to invoke tests via pytest or python -m unittest. Create folder common/tests/unittests containing unittests only. With this solution a make test will invoke all tests (including the ones in the sub-folder tests/unittests) and make unittest will invoked only the tests in the sub-folder. This should be easy to realize via our configure script. More accurate we also could create tests/system and tests/integration which would force us to think about what test kind we do have. Another side effect would be a more realistic coverage number. IMHO "code coverage" should count only for unittests.

The usual make unittest then could be used by Debians build system (see below).

More background

Related to the Debian bug ticket #940319 I did some more research and also got some valuable advice of people around Debians Python Packaging Team. The Debian ticket is about the problem that our tests want to write to HOME folder which is not possible in Debians build system because they do chroot as a sandbox solution.

But fixing the HOME problem with using a temp folder or a fake in-memory-filesystem won't solve the problem described in this Issue here. Our tests do create ssh keys, installing them, contacting an ssh server, etc. There is no way to mock that and there is no need for it. Integration and system tests are also valuable and they still prevented us from some mistakes.

Which tests to separate?

Looking through the code in common/tests I only can find diagnostics.py to be a real unittest without dependencies to other BIT components. Also argparse.py is a candidate for a unittest but need some more research and modificaton. All other tests seem to be real integration- or system-tests to me. No other unittests do exist.

Sorry, this issue is not fun. 😿

buhtz commented 10 months ago

Related to #1463

buhtz commented 10 months ago

I tried to fix this with buhtz:1489testcategories but messed up with TravisCI, Coveralls and configure-make script I'm not familiar with.

My original proposal still make sense to me and should be the goal. But I will make a pause from here on that Issue. It should wait until we migrated to modern Python packaging standards (slightly discussed in #1513). After that the build environment and imports are way easier to handle.

buhtz commented 6 months ago

Depends on #1575