purebred-mua / purebred

A terminal based mail user agent based on notmuch
GNU Affero General Public License v3.0
139 stars 19 forks source link

UAT concurrency #422

Closed frasertweedale closed 3 years ago

frasertweedale commented 3 years ago
06002f6 (Fraser Tweedale, 12 minutes ago)
   uat: use separate config dir for each test

   With test concurrency, some tests are interfering with each other. In
   particular, the "send mail" action we use for testing writes the send mail
   to a static filename in the config directory.

   Update the test suite to create a separate config dir for every test.  The
   config and precompiled binary are copied into each test's config directory
   during setup to avoid recompilation.

   I did consider hard-linking instead of copying.  This would avoid copying
   the (quite large) purebred custom binary to each test's config dir.  But I
   decided against this.  Future tests might modify the configuration and we
   need to keep it "pristine" for each test.

3fdd83b (Fraser Tweedale, 61 minutes ago)
   uat: use concurrency to speed up execution

   tasty recognises the `TASTY_NUM_THREADS` environment variable.  Set this
   environment variable (or leave it alone if already set) so that tests go
   faster.

   I experimented on my machine and found that the value `20` gives good
   results.  Using more threads does not speed things up much, and makes test
   failures highly likely (with 40 threads, more likely to fail than succeed).
    Failures are already more common with 20 threads, but the test suite
   finishes faster so it is a reasonable tradeoff.

   I understand some of the failures already.  Fixes will follow in subsequent
   commits.  Other failures need investigation.