06002f6 (Fraser Tweedale, 12 minutes ago)
uat: use separate config dir for each test
With test concurrency, some tests are interfering with each other. In
particular, the "send mail" action we use for testing writes the send mail
to a static filename in the config directory.
Update the test suite to create a separate config dir for every test. The
config and precompiled binary are copied into each test's config directory
during setup to avoid recompilation.
I did consider hard-linking instead of copying. This would avoid copying
the (quite large) purebred custom binary to each test's config dir. But I
decided against this. Future tests might modify the configuration and we
need to keep it "pristine" for each test.
3fdd83b (Fraser Tweedale, 61 minutes ago)
uat: use concurrency to speed up execution
tasty recognises the `TASTY_NUM_THREADS` environment variable. Set this
environment variable (or leave it alone if already set) so that tests go
faster.
I experimented on my machine and found that the value `20` gives good
results. Using more threads does not speed things up much, and makes test
failures highly likely (with 40 threads, more likely to fail than succeed).
Failures are already more common with 20 threads, but the test suite
finishes faster so it is a reasonable tradeoff.
I understand some of the failures already. Fixes will follow in subsequent
commits. Other failures need investigation.