make -j <num> check failures

[hdatma]

Subsequent runs of "make check" return different results. Sometimes is passes all test, sometimes it fails the following.

> grep FAIL test.log                                                                                                          
FAIL: t-test93
FAIL: t-test99
FAIL: t-test88
FAIL: t-test84
FAIL: t-test94
FAIL: t-test89
FAIL: t-test78
FAIL: t-test73
FAIL: t-test72
FAIL: t-test67
FAIL: t-test68
FAIL: t-test69
FAIL: t-test82
FAIL: t-test80
FAIL: t-test48
FAIL: t-test64
FAIL: t-test53
FAIL: t-test46
FAIL: t-test56
FAIL: t-test43
FAIL: t-test51
FAIL: t-test38
FAIL: t-test41
FAIL: t-test45
FAIL: t-test32
FAIL: t-test40
FAIL: t-test35
FAIL: t-test36
FAIL: t-test29
FAIL: t-test30
FAIL: t-test34
FAIL: t-test31
FAIL: t-test23
FAIL: t-test21
FAIL: t-test22
FAIL: t-test28
FAIL: t-test20
FAIL: t-test15
FAIL: t-test149
FAIL: t-test153
FAIL: t-test37
FAIL: t-test138
FAIL: t-test142
FAIL: t-test134
FAIL: t-test139
FAIL: t-test136
FAIL: t-test13
FAIL: t-test126
FAIL: t-test117
FAIL: t-test125
FAIL: t-test14
FAIL: t-test12
FAIL: t-test114
FAIL: t-test39
FAIL: t-test120
FAIL: t-test102
FAIL: t-test07
FAIL: t-test04
FAIL: t-test06
FAIL: t-test05
FAIL: t-test123
FAIL: t-test101
FAIL: t-test115
# XFAIL: 0
# FAIL:  63

The failures occur only when running parallel tests, for example, "make -j 12 check".

[orlitzky]

This is also https://bugs.gentoo.org/745969 and Francisco Blas Izquierdo Riera (klondike) figured out what's going wrong:

Between https://github.com/trusteddomainproject/OpenDKIM/blob/develop/libopendkim/tests/Makefile.am#L20 and https://github.com/trusteddomainproject/OpenDKIM/blob/develop/libopendkim/tests/Makefile.am#L51, a list of tests is constructed. The first one is the "setup" test, and the last one is the "cleanup" test. The build system expects the setup test to be run first, and the cleanup test to be run last; if not, you get test failures due to missing keys.

Since make -j<num> check runs all of the test programs in parallel, the execution order is not guaranteed. Either the setup/cleanup tests should be forced to run first/last somehow, or something else should be done to ensure the availability of the keys. The solution here should also address https://github.com/trusteddomainproject/OpenDKIM/issues/113.

[thegushi]

"somehow" is really annoying here. Other than reverting to the serial test harness with automake (warning, don't use, deprecated), etc, it seems that:

• Yes, you can define a test as a prerequsite of another by declaring their logfiles as targets, eg: t-01.log: t-setup.log

• The test harness has no specific support for "run first" or "run last" commands. It also doesn't have a "run this before your suite" command. There's only AM_TESTS_ENVIRONMENT, which runs on EVERY test.

• There's not an easy (portable way) that I can find in GNU Automake to say "prerequisites for t-cleanup.log are, literally, the logfiles of every other test in check_programs" without defining it explicitly. Any attempts to build the list programatically throw autoconf warnings.

• I could use AM_TESTS_ENVIRONMENT to run an init stage, ONLY IF the keys that are required are not present.

• On the same note, the "cleanup" stage does nothing but delete the keys, which if we're going to move them out of /tmp, maybe we should just statically define them and call it good. The t-setup script really does nothing other than copying some key material somewhere, and t-cleanup just rm's it.

[orlitzky]

How about,

1. Drop t-cleanup entirely.
2. Remove t-setup from TESTS.
3. Have each test that needs the keys depend on them in a fixed location under the build directory.
4. Add a Makefile.am target for the keys that depends on (and runs) the t-setup program to create them.
5. Finally, add the keys to CLEANFILES so that make clean does what t-cleanup used to do.

Step 3 is a little annoying, but not all of the tests require keys, so listing them will help it parallelize. And this should fix the /tmp issue too.