Test suite failure running latest nightly

[snoyberg]

asymptotic perf roundtrips
  1000 do statements FAILED [1]
  1000 do nestings
  1000 AppOps

Failures:

  src-unittests/TestUtils.hs:32:43: 
  1) asymptotic perf roundtrips 1000 do statements
       predicate failed on: Nothing

  To rerun use: --match "/asymptotic perf roundtrips/1000 do statements/"

Randomized with seed 1126456497

Finished in 3.9674 seconds
3 examples, 1 failure

This appears to be a new failure with 0.12.

[lspitzner]

Is this result consistent? Not sure if this is apparent, but the failing test does depend on a timeout, so you might get spurious failures if your system is busy or slow in general. If it is consistent, can you give any details on the environment?

I have manually tested this, and cannot spot any difference in performance between 0.11 and 0.12. The only interesting change in my testing was that with ghc-8.6, the "memory in use" has dropped from 12MB to 4MB.

I concede that testing a non-functional feature like this is somewhat sloppy. The idea was that the timeout was large enough that you'd only ever get failures if the asymptotic complexity had changed, and even if this is more of a very rough benchmark, it has worked well so far (even on the CI servers that are a bit less reliable I have not run into problems).

(I wonder if it is worth tracking performance in a disciplined fashion, e.g. in CI somehow. This is just a formatter, not a compiler, but it should not be that hard to set something up either.)

[snoyberg]

I didn't look closely, and hadn't noticed that it was a performance issue. We run multiple tasks on our build server in parallel, which might explain this, I'm not sure. I'll try reenabling the tests for now. Feel free to close this issue if you're happy with things as-is.