Currently tool tests are useful as a developer tool - to ensure tools work correctly with a default job runner and default dependency resolution - but the work that goes into them could be made useful for other larger target audiences as well.
Administrators
We should provide an API endpoint for execute tool tests.
[ ] Move more of the test driving logic into Galaxy core proper.
[ ] Provide and expose abstractions for discovering test data location for configured tools.
Such an API could then be executed with cron jobs, as part of deployment scripts, as part of upgrade procedures, etc... to ensure both that the tool is still working but also checking job runners and the infrastructure they plug into as well as dependency management configuration and the infrastructure they plug into.
Once the API endpoint is created it would be nice to add both an Admin UX for these as well as an ephemeris script for driving them.
Users
Users can now run tool tests as part of https://github.com/galaxyproject/galaxy/pull/4019. I think enhancing both that webhook as well the Galaxy tool format we could make tool tests much more informative to the end user and something ultimately a lot more educational. Ideas for doing this include:
[ ] Annotations on tests that explicitly describe the test case as a whole and on particular parameters as needed. These things should be presented to users.
[ ] Only display test cases with annotation - we have test cases that test things fail or instance - they shouldn't be exposed to end users.
[ ] Extend TDT to work with repeats, collections, etc...
[ ] Add a formal API endpoint or abstractions on the tool class in the code to find the test data (part of the above topic).
[ ] Moved out of the webhook framework and into the compiled code base.
Currently tool tests are useful as a developer tool - to ensure tools work correctly with a default job runner and default dependency resolution - but the work that goes into them could be made useful for other larger target audiences as well.
Administrators
We should provide an API endpoint for execute tool tests.
Such an API could then be executed with cron jobs, as part of deployment scripts, as part of upgrade procedures, etc... to ensure both that the tool is still working but also checking job runners and the infrastructure they plug into as well as dependency management configuration and the infrastructure they plug into.
Once the API endpoint is created it would be nice to add both an Admin UX for these as well as an ephemeris script for driving them.
Users
Users can now run tool tests as part of https://github.com/galaxyproject/galaxy/pull/4019. I think enhancing both that webhook as well the Galaxy tool format we could make tool tests much more informative to the end user and something ultimately a lot more educational. Ideas for doing this include: