Project: Run tests on tools updates

[jmchilton]

• [x] Update planemo to allow tool testing against external servers.
• [x] Develop tooling to automate this as part of ephemeris updates.
• [x] usegalaxy-eu integration @erasche
  • [x] test usegalaxy-eu tools (https://github.com/usegalaxy-eu/usegalaxy-eu-tools).
  • [x] test usegalaxy-eu worflows

[hexylena]

Is there any possibility we could implement workflow testing while we're there? https://github.com/galaxyproject/galaxy/projects/8#card-7262571

EU really desperately wants this and we'll help however we can. (Need a crate of your favourite beverage/brand of hash browns? We can do this! Need a live popular server to test against? Done.) We would really love to be able to prove that all of the training workflows will run successfully without issue on our server.

[rhpvorderman]

@jmchilton can this issue be closed? You successfully enabled tool testing from the command line.

[hexylena]

We haven't ticked the last box yet @rhpvorderman (or, second to last now that I've re-organised it.) We'll get there soon hopefully.

[hexylena]

I tried to run planemo test --engine galaxy tools-iuc/tools/jq.xml yesterday but it failed to load test data, it seemed to try and load that from the server path and not my local path.

[mvdbeek]

Did you do it with an admin key ? AFAIK we didn't lift that requirement yet.

[hexylena]

Yup, did it under my admin account.

[hexylena]

urllib3.connectionpool DEBUG 2018-08-06 16:08:43,058: https://usegalaxy.eu:443 "GET /api/tools/jq/test_data_path?filename=list.json&key=<redacted> HTTP/1.1" 200 None
[Errno 2] No such file or directory: '/usr/local/galaxy/shed_tools/toolshed.g2.bx.psu.edu/repos/iuc/jq/5ff75eb1a893/jq/test-data/list.json'
Traceback (most recent call last):
  File "/home/hxr/arbeit/planemo/.venv/bin/planemo", line 11, in <module>
    load_entry_point('planemo', 'console_scripts', 'planemo')()
  File "/home/hxr/arbeit/planemo/.venv/lib/python3.5/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/hxr/arbeit/planemo/.venv/lib/python3.5/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/hxr/arbeit/planemo/.venv/lib/python3.5/site-packages/click/core.py", line 1066, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/home/hxr/arbeit/planemo/.venv/lib/python3.5/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/hxr/arbeit/planemo/.venv/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/hxr/arbeit/planemo/.venv/lib/python3.5/site-packages/click/decorators.py", line 64, in new_func
    return ctx.invoke(f, obj, *args[1:], **kwargs)
  File "/home/hxr/arbeit/planemo/.venv/lib/python3.5/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "/home/hxr/arbeit/planemo/planemo/cli.py", line 195, in handle_blended_options
    return f(*args, **kwds)
  File "/home/hxr/arbeit/planemo/planemo/commands/cmd_test.py", line 79, in cli
    test_data = engine.test(runnables)
  File "/home/hxr/arbeit/planemo/planemo/engine/interface.py", line 79, in test
    test_results = self._collect_test_results(test_cases)
  File "/home/hxr/arbeit/planemo/planemo/engine/interface.py", line 98, in _collect_test_results
    run_response = self._run_test_case(test_case)
  File "/home/hxr/arbeit/planemo/planemo/engine/galaxy.py", line 89, in _run_test_case
    return test_results[0]
IndexError: list index out of range

[mvdbeek]

So shed-tools seems to work fine with this invocation

shed-tools test -g http://<GX_IP>/ -a <MY_API_KEY>

where GX_IP is remote. I guess the planemo implementation still needs some work

[mvdbeek]

It does skip a lot of repos to tests though, maybe that's those that actually require datasets ?

[hexylena]

TIL shed-tools test instead of planemo test --engine galaxy --galaxy_url ...

[hexylena]

Thanks @martenson