Import libcperciva test framework; use for lbs

[gperciva]

This begins the process of moving the existing test suite into the libcperciva-style tests.

test command

Normally, we'd run this with

make test

However, kivaloo already has a considerable set of tests, and we don't want to disable those until the new test framework covers at least that much.

As a temporary measure, the new tests can be run with:

make test-libcperciva-framework

test verbosity

The old tests were more verbose; for example,

$ ./test_lbs.sh 
Testing LBS operations... PASSED!
Testing LBS connection-closing cleanup... PASSED!
Testing LBS disconnection cleanup... PASSED!
Testing LBS connection queuing... PASSED!
Testing LBS with socket at localhost:1234... PASSED!
Testing LBS with socket at [127.0.0.1]:1235... PASSED!
Testing LBS with socket at [::1]:1236... PASSED!
Checking for memory leaks in LBS...38 allocations leaked 22496 bytes
  -> memory leaks shown in lbs.leak
Checking for memory leaks in LBS client code...2 allocations leaked 1184 bytes
  -> memory leaks shown in test_lbs.leak

The equivalent in this PR is:

$ make test-libcperciva-framework
... (compiling)...
Running tests
-------------
  01-lbs-basic... SUCCESS!

I could split up 01-lbs-basic into 01- to 07-, if you wanted. I don't think's necessary, and we might run out of two-digit numbers if we split up all the tests.

Memory checking

For memory checking, the new suite uses valgrind (as with other libcperciva). The old test suite uses ktrace and kdump on FreeBSD, and nothing on other platforms.

(As seen above, ktrace reports leaks in both lbs and test_lbs, but I think those are false positives -- actually, the biggest reason I started working on the new test suite was so that I could use valgrind, which already has a whole "suppress system-library leaks / known false positives" system.

I think that losing ktrace wouldn't be a problem, but if it is, I could import that to the new system as well.

Other

I thought there was at least another major point to mention, but it's not coming to mind.

As far as the commits go, the first one is trivial importing stuff; the only important stuff is in the second commit.

[cperciva]

LGTM. Just to confirm before I merge this: If the test fails will we get an indication of how it failed, not just "01-lbs-basic... FAILED!"? That's the only reason my original test script was more verbose -- to tell me which of the many parts had failed.

Valgrind is far better than my hacky ktrace+parsing approach so I'm happy for the ktrace bits to go away now that you have valgrind.

[gperciva]

No, the libcperciva test suite doesn't print failure messages on the console.

There's additional info in the tests-output/ directly; in particular, any test that exits with non-zero will leave a file, such as

$ ls tests-output/
tests/output/01-lbs-basic-12-lbs-server.desc
tests/output/01-lbs-basic-12-lbs-server.exit
$ more tests-output/01-lbs-basic-12.desc 
test_lbs check localhost:1234

This limitation has occasionally been annoying when trying to debug something that fails in github actions but not on my default development system; but so far I've always been able to reproduce those cases in virtualbox.

I could expand the test suite framework to be more verbose on failure?

[cperciva]

That's fine, just wanted to make sure it was going somewhere.

Maybe worth adding an option to print the extra information and turn that on in the github actions?

[gperciva]

Huh, I wrote better code than I thought! The code is already there; all we have to do is set VERBOSE=1.

[gperciva]

Example from a deliberate failure in libcperciva with VERBOSE=1:

Running tests
-------------
  01-crc32... SUCCESS!
  02-getopt... SUCCESS!
  03-heap... SUCCESS!
  04-humansize... SUCCESS!
  05-monoclock... SUCCESS!
  06-parsenum... FAILED!
File /Users/runner/work/libcperciva/libcperciva/tests-output/06-parsenum-00.exit contains exit code 1.
Test description: DEBUG deliberate failure