[feature request] Export workflow invocation as planemo workflow test

[hexylena]

Ideally just like the RO-Crate and BCO export options, we could have a "Planemo workflow test" export that'd be a bit more friendly than having to run planemo manually just to export the skeleton test case

From a workflow invocation you'd get something like:

.
├── test-data
│   ├── input dataset(s).shapefile.shp
│   └── shapefile.shp
├── testing-openlayer.ga
└── testing-openlayer-tests.yml

and then the user could further edit the test cases.

[mvdbeek]

Doing this an an export is a cool idea! With how things are now the tricky part is finding a way to isolate the planemo dependency, which might well conflict with Galaxy.

[nomadscientist]

Oh my god this would be amazing. Workflow testing has been a major blocker for us, bless your cotton socks

[mvdbeek]

This sure would be an improvement, but I would love to hear about the blockers you have encountered.

[nomadscientist]

Oh sure - it's delayed us submitting tutorials by 1 month+ because we struggle to get workflow testing to work. @MarisaJL can you add anything here?

[mvdbeek]

Can you open a new issue for this at https://github.com/galaxyproject/planemo/issues ?

[nomadscientist]

I can, but the issue would read for me personally as follows:

• Can't get Planemo to run at all on my Mac.
• Can't get virtual environment to run.
• Planemo is not usable unless you have reasonable programming skills beyond applied analysis
• This is a bit much for non-developers as a barrier to building useful analysis workflows for bench biologists
• Please don't make me have to use Planemo.

[nomadscientist]

Here's a play-by-play: https://training.galaxyproject.org/training-material/faqs/gtn/gtn_workflow_testing.html Instruction:

# In a virtualenv
pip install planemo

Me: Googles 'how to set up a virtual env' Result: https://sourabhbajaj.com/mac-setup/Python/virtualenv.html Instruction pip install virtualenv Result: pip install virtualenv -bash: pip: command not found

Me: Googles 'pip: command not found' Result: https://www.freecodecamp.org/news/pip-command-not-found-mac-and-linux-error-solved/ Instruction: python3 -m pip --version Result: pip 23.2.1 from /Users/wb2845/Library/Python/3.8/lib/python/site-packages/pip (python 3.8) Instruction: python -m pip install --upgrade pip Result: -bash: python: command not found

Me: Stuck

[mvdbeek]

Please don't make me have to use Planemo.

What do you think happens after you generate a test skeleton ? You're gonna edit the test assertions in word ?

At this point we do require command line literacy for contributing to the GTN, I don't think we're quite at the point where this is so streamlined that we can offer this for workflow tests.

Me: Stuck

Have you considered asking for help, and/or i don't think workflow tests are currently mandatory in the GTN.

[hexylena]

don't think workflow tests are currently mandatory in the GTN.

they are, since I think around spring this year.

[mvdbeek]

I hate to say it, because I think it's awesome that you do that, but that seems like quite a barrier to entry ? I would say writing tests does require a good understanding of test assertions and workflows in general, and you can't do tests on e.g. composite inputs. The test skeleton is nice, but I don't think it would pass tests in practice without tweaking them ?

[hexylena]

You're not wrong it's definitely a barrier, a big one at that. It's an issue for sure, unfortunately.

We really want it, since it'll give us a lot of benefits that even (people who experience said barriers) are explicitly asking for (e.g. example histories, automated testing against .*) Those asks from the users for these features motivated us adding this barrier even though it's gone and negatively affected them directly.

I'm not sure what the best way to solve that is, but I suspected that this issue itself would be a big step in the right direction. It'd let those users go back to working on their workflows in galaxy, tutorials in word/notepad/etc (yes this is a thing), and submitting PRs via the GitHub upload/edit UI. Their workflow would be "make your workflow, run it once, export invocation, export tutorial skeleton, edit those two files."

It's forcing folks to check workflow best practices (good, lower barrier) but overall the result is quite mixed.

but I don't think it would pass tests in practice without tweaking them ?

no, essentially all tests need modifications, we link to the planemo documentation but probably our users could use some more explicit documentation focused on them (based on their discipline/skill level with these topics), but we hadn't received any feedback on the issues there yet, possibly due to the python/pip/venv, pre-planemo barriers

and you can't do tests on e.g. composite inputs

thankfully rare in the training case, but, good to know for when it comes up, cheers!

[mvdbeek]

It'd let those users go back to working on their workflows in galaxy,

How about we make the invocation itself sharable and generate just a job input object, not a test, in a github action ? Then the users really just need to run the workflow once, and if you want you can always add tests later ? This seems to be more about workflows being runnable than making specific assertions ?

[mvdbeek]

Or, we require an ro-crate of the invocation, from which we can also generate tests ... that wouldn't need any backend changes.

[hexylena]

if we can generate tests from an RO-Crate that would be an easy win, that sounds workable

[hexylena]

But I guess at that point we have the result of

flowchart LR;
  galaxy --> ro-crate --> ? --> test-case;

and that ? needs to be filled with something? I guess something like https://ptdk.apps.galaxyproject.eu to give the users a web-interface to convert their invocation objects, which, we could do just as easily from the invocation itself, and I'm not sure the RO-Crate wins us anything there (since we don't hvae a good mechanism for them e.g. adding an RO-Crate to their PR, and then automatically updating that PR with a test case extracted from that.)

that said I'd still love to see this issue and a workflow/api like the following in the future, if it's possible long term.

flowchart LR;
  galaxy --> test-case;

[mvdbeek]

Why do you want tests ? Isn't it sufficient to make sure the workflow runs ? for that you just need an input object and no manual tweaks required

since we don't have a good mechanism for them e.g. adding an RO-Crate to their PR, and then automatically updating that PR with a test case extracted from that.)

can't you just let them stick in a url to the ro-crate in the tutorial or the PR description and do the rest as usual with planemo in an action ?

that said I'd still love to see this issue and a workflow/api like the following in the future, if it's possible long term.

sure, I suggested the ro-crate because that's actually sharable, which invocations are not

[hexylena]

Isn't it sufficient to make sure the workflow runs ? for that you just need an input object and no manual tweaks required

No, unfortunately. We want a real test, we want to be explicit, and I think in a lot of cases people will be able to use has_lines which are pretty easy to add and let us ensure that the workflows are still producing the correct outputs over time and over different servers. It'll make it easier to onboard new servers when we host a Smörgåsbord if we can be certain the jobs aren't just completing but have the right output.

can't you just let them stick in a url to the ro-crate in the tutorial or the PR description and do the rest as usual with planemo in an action ?

ah github actions engineering. That's,, yeah, that's an option. It's not my favourite solution since people already completely ignore our PR template, but it could work for repositories which we can commit to. (There's a couple contributors that do not permit us to push back any commits due to organisational settings.)