Fix CI

[stevana]

• CrashAndLogicBugParallel sometimes fails, even with maxSuccess 3000...
• Builds timeout often (perhaps because a new release of stack invalidated the old cache)
• Generally more flaky since Travis got bought up

Possible solutions:

• Make a "fast test suite" that doesn't run all tests? Is the build too slow or does running the tests take too long?
• Disable tests temporarily, so that cache gets built, then enable tests again?
• Remove stack builds from CI?
• Investigate alternatives to Travis?

Thoughts?

[kderme]

I thinks it's some tests which take too long. I can try a couple things to make them faster.

[stevana]

Here's another problem that I've not been able to reproduce locally:

 NoRaceBug:                                                    FAIL
      Exception: QuickCheck.elements used with empty list
      CallStack (from HasCallStack):
        error, called at ./Test/QuickCheck/Gen.hs:203:15 in QuickCheck-2.13.1-1rO57X9FmbTEL9U6cBekOs:Test.QuickCheck.Gen

(Could't find any seed or --quickcheck-replay to reproduce this in the logs, shouldn't that always be there?)

[kderme]

Is this the correct link you have above? I couldn't find the failure..

[stevana]

I'm pretty sure (I doubled checked when I posted), could it be that because I reran the job (because it timed out) it overwrote the logs...?

[stevana]

Hmm, https://travis-ci.org/advancedtelematic/quickcheck-state-machine/jobs/553776901:

...
Setting up build cache
$ export CASHER_DIR=${TRAVIS_HOME}/.casher
0.16s$ Installing caching utilities
0.00s1.18sattempting to download cache archive
fetching master/cache-linux-trusty-811a73447d636739ee8e0adc0d376798ed2e3294169c24b7738557e7b2e4b580.tgz
could not download cache
0.00s1.07sadding /home/travis/.ghc to cache
creating directory /home/travis/.ghc
adding /home/travis/.cabal to cache
creating directory /home/travis/.cabal
adding /home/travis/.stack to cache
creating directory /home/travis/.stack
adding /home/travis/build/advancedtelematic/quickcheck-state-machine/.stack-work to cache
creating directory /home/travis/build/advancedtelematic/quickcheck-state-machine/.stack-work

and then a bit later, stack says:

No old cache found, populating cache from scratch

[stevana]

Caching of stack build seems to work again now (no change made).

[stevana]

The Overflow 4 thread property sometime fails on CI:

https://travis-ci.org/advancedtelematic/quickcheck-state-machine/jobs/553916088

When I tried to reproduce it locally it first failed, but not for the right reason:

      *** Exception running callback: 
      dot: runInteractiveProcess: runInteractiveProcess: exec: does not exist (No such file or directory)

Again bitten by how expectFailure thinks exceptions are failures...

Anyway, after installing graphviz, renaming testProperty "Overflow 4 threads" to testProperty "4-threads" and running:

stack test --ta="-p Overflow.4-threads --quickcheck-replay=481701"

(The reason for renaming the property is that I don't know how to run properties with spaces in the name with the -p flag.)

I got:

Tests
  Overflow
    4-threads: FAIL (27.30s)
      *** Failed! Passed 100 tests (expected failure).
      Use --quickcheck-replay=481701 to reproduce.

1 out of 1 tests failed (27.30s)

I guess we need to fix three things:

• [x] Rename properties so we can run them separately
• [x] Higher maxSuccess for that property
• [x] Check if dot is in PATH, if not prompt the user to install it and exit gracefully
• [ ] Install graphviz on Travis?

[kderme]

Sorry for creating this issue with dot

Install graphviz on Travis?

Do you think this is possible?

[stevana]

Yeah, should be an apt-get install graphviz somewhere in the Travis config.

But, we probably want to avoid crashing when dot is not available and we will probably never look at the .svg produced on Travis. So maybe we don't need to actually install graphviz? It should just say, "Found counterexample, but cannot visualise it because graphviz isn't installed" -- I think that would be good enough for CI?

[kderme]

Yes, you're right. I'll write some handlers for the case where graphviz is not install.

[kderme]

I'm currently writing a test for rqlite, which may require installing https://github.com/rqlite/rqlite/releases/tag/v4.5.0 if we want to run it on ci. Do you think that's possible?

[stevana]

It's possible to install anything on Travis, but perhaps using docker is easier?

[stevana]

(Like we do with postgres, etc. Because then anyone with docker installed can run our test suite, and we already skip the tests which require docker if docker isn't installed.)

[stevana]

The following CI run, https://travis-ci.org/advancedtelematic/quickcheck-state-machine/jobs/560407777 , contains a new problem:

    Bookstore input validation bug:                                
FAIL (20.01s)
      *** Failed! Passed 100 tests (expected failure).
      Use --quickcheck-replay=361731 to reproduce.

Need to bump maxSuccess...

[stevana]

I think this is fixed, for now...