Testing Roadmap for 2016

[jmchilton]

16.04

• [ ] Fix naming and versioning of testing Galaxy testing Docker image for this release.
• [x] Sniffer tests (could be done using API tests or integration tests).
• [x] Python integration tests. These are lighter than API tests since they will create an app but not a server and allow modifying config/ but heavier than unit tests since they will initialize most Galaxy subsystems.
• [x] Document all GALAXY_TEST_ options somewhere.
• [x] Implement (or re-implement) testing against running Galaxy servers. Tool tests should be runnable as well as a subset of API tests for instance (perhaps annotating each test which config options that must be set a certain way).

  16.07

• [ ] Fix bugs with running casperjs tests in Docker on Jenkins.
• [ ] Re-enable reporting of casperjs test results.
• [ ] Start collecting coverage test information for Travis and send to coveralls.io.

  16.10

• [ ] Extend coverage test collection to API and framework tests on Jenkins.
• [ ] Create a Galaxy testing Docker image for this release.

  Other Tasks

• [ ] Consider running latest stable BioBlend and planemo tests against each Galaxy PR and reporting the results since they will catch their own set of issues.

[bgruening]

Consider running latest stable bioblend and planemo tests against each Galaxy PR and reporting the results since they will catch their own set of issues.

The Galaxy Docker image does exactly this every two days.

• Build the Docker image
• run all bioblend tests against it
• test nginx and herewith the ansible roles
• test FTP Server

Let me know if we can take something from this back to Galaxy main.

[jmchilton]

Re-arranged things to reflect what was actually done for 16.04 and pushed some stuff off to 16.07 unfortunately. Created a more specific issue for coverage testing (https://github.com/galaxyproject/galaxy/issues/2081).