Invoking interop jobs automatically

[trungdong]

Investigate a way to invoke the rerunning of a test job automatically. This can happen, say, when a ProvPy build on Travis is successful. Then, the provpy-interop-job will automatically be run.

The ProvPy repo page, for example, can include another icon for the status of its interop tests and a developer can be aware of the existence of any interop issue without having to invoke the interop tests manually.

[mikej888]

A test job could be rerun automatically via use of the Travis Client (as described in Travis Client). If the ProvPy build succeeds, it could invoke the travis restart command to restart the associated interoperability tests.

I implemented a simple example of this in travis-sender and travis-receiver. travis-receiver has a Travis CI job that just prints a message. travis-sender has a Travis CI job that uses the Travis Client:

language: python

python:
  - 2.7

install:
  - gem install travis

script:
  - echo "Hello"
  - ruby -v
  - travis version --skip-completion-check
  - travis login --skip-completion-check --github-token $GITHUB_TOKEN
  - travis history --skip-completion-check -r mikej888/travis-receiver
  - travis restart --skip-completion-check -r mikej888/travis-receiver
  - travis logout --skip-completion-check

The value of GITHUB_TOKEN being a hidden environment variable configured via the Travis CI user interface. The --skip-completion-check flag is (unfortunately) required otherwise travis pops up an interactive prompt asking Shell completion not installed. Would you like to install it now? |y|.

Every time this job is run, it triggers a re-run of the last travis-receiver job. See the Travis CI jobs for:

• travis-sender
• travis-receiver

However, I don't think this approach should be used in practice, since it becomes increasingly complex for third-party developers to set up to run their own tests. They would need to:

• Clone ProvPy Git repository on GitHub.
• Enable this under Travis CI.
• Configure Travis CI for this job with their GitHub API key.
• Clone ProvPy interoperability test job Git repository on GitHub.
• Enable this under Travis CI.
• Update ProvPy .travis.yml with the name of their Travis CI job (e.g. mikej888/provpy-interop-tests)
• Update ProvPy interoperability test job .travis.yml with the location of their ProvPy repository.

This introduces a circular dependency between the two repositories. It also makes submitting pull requests back to the canonical ProvPy repository more difficult, as users have to restore the default .travis.yml file before doing so (which they may forget to do).

More generally, I've been thinking about the use of Travis CI to run interoperability tests via Travis Client, and I'm not so sure this is a good idea, as it is somewhat of an abuse of the concept of Travis CI (that is, tests are rerun every time a change is committed). It would be more in the spirit if the interoperability tests for ProvPy, for example, were rerun every time a change to ProvPy was committed (but without the convoluted approach above). This relates to #11.