search

qiime2 / galaxy-tools

Official QIIME 2 tools for Galaxy
BSD 3-Clause "New" or "Revised" License
1 stars 4 forks source link

Figure out how test toolsheds work #10

Open ebolyen opened 2 years ago

ebolyen commented 2 years ago

We have a plugin called q2-mystery-stew which produces the tests used by q2galaxy.

Ideally we could render this plugin and start testing toolshed server to install this suite and run the tests. This would also help prove that our .shed.yml files are meaningful.

There's probably a lot of overlap here with #7

ebolyen commented 2 years ago

A question, is there any conventions for a tools directory which is either only for testing, or is somehow excluded from uploading?

I would prefer Galaxy users not interact with q2-mystery-stew directly on the toolshed, as it's literally a combinatoric soup of nonsense.

bernt-matthias commented 2 years ago

The CI implemented in https://github.com/galaxyproject/galaxy-tool-repository-template (which is used at IUC and essentially every other Galaxy tool repository) currently does not allow to skip deployment of certain tools. But this should be easy to implement.

Ping me if you think this is needed.

start testing toolshed server to install this suite and run the tests

Note that the toolshed is not running any tool tests. Tool tests of all repos run via github actions:

Furthermore there is the possibility to run slash commands in issues to test against certain Galaxy branches (which might be nice to check tools against a Galaxy PR, e.g. an upcoming datatype addition PR)

ebolyen commented 2 years ago

Another option might be to have a parallel test branch with an alternate CI configuration, so that upload is avoided, but tests still run.

These github actions are really detailed! It does look like it uses planemo test under the hood. This means there isn't some mock toolshed which a galaxy instance is fetching from and then testing? If that's the case, then testing with mystery-stew can probably just continue to live in q2galaxy and we don't need to do anything special in this repo. This is what we do at the moment.

Then this particular repo can look like any of the other ones.

bernt-matthias commented 2 years ago

Another option might be to have a parallel test branch with an alternate CI configuration, so that upload is avoided, but tests still run.

We already have a files .tt_skip and .tt_biocontainer_skip to control tools / directories that should not be subject to CI or should be tested without containers (funnily good old qiime1 is in there https://github.com/galaxyproject/tools-iuc/blob/4e41978ca2530441bd5abf07a37ab9e8eedfce6c/.tt_biocontainer_skip#L2). So it should be really easy to setup another one to forbid deployment. But another branch also sounds useful .. up to you :)

These github actions are really detailed! It does look like it uses planemo test under the hood.

Indeed. Everything works with this action https://github.com/galaxyproject/planemo-ci-action which uses planemo.

This means there isn't some mock toolshed which a galaxy instance is fetching from and then testing?

Exactly.

If that's the case, then testing with mystery-stew can probably just continue to live in q2galaxy and we don't need to do anything special in this repo. This is what we do at the moment. Then this particular repo can look like any of the other ones.

Sound reasonable. Guess the tool tests derived from Usage Examples will end up here, or?

ebolyen commented 2 years ago

Sound reasonable. Guess the tool tests derived from Usage Examples will end up here, or?

Yep, any examples that do exist (with ideally more in the future) will become test sections and should be executed here.