EOSC Life Monitor

[ewels]

Placeholder issue to remind us about the EOSC Life Monitor service: https://www.researchobject.org/ro-crate/in-use/life_monitor

As I understand it, you use Workflow RO Crates (https://github.com/nf-core/tools/issues/686) to describe where you are running CI tests, and EOSC Life Monitor then aggregates these and monitors them and stuff.

• RO-Crate docs for this: https://www.researchobject.org/ro-crate/profiles.html?asdfasdf#workflow-testing-ro-crate-profile
• Wiki docs: https://github.com/crs4/life_monitor/wiki/Workflow-Testing-RO-Crate

[simleo]

• Metadata spec: https://github.com/crs4/life_monitor/wiki/Test-Metadata-Draft-Spec

That's the old version of the specs, still there for reference. Can you please delete that bullet? Sorry for the confusion.

[simleo]

Thanks for opening this @ewels. Adding some more info.

Yes, Life Monitor is a workflow monitoring and testing service under development as part of EOSC-Life.

Part of the work is the development of an RO-Crate extension to allow the specification of test suites associated with a workflow. Currently this involves the monitoring of existing test runs on CI instances, but another planned functionality is running tests according to a definition (also described via the RO-Crate metadata), with the aim of leveraging existing workflow testing engines. Workflows to be monitored are expected to come primarily from the Workflow Hub, also developed within EOSC-Life and supporting RO-Crates.

For instance, Galaxy workflows can be tested with Planemo, so a workflow author/maintainer can describe tests for a Galaxy workflow via a Planemo YAML file specifying inputs and expected outputs, and reference it from the RO-Crate metadata according to the above specs.

So one of the things we're wondering is how one would go about that for a Nextflow workflow. Are there any plans for a testing engine (i.e., something similar to Planemo) for Nextflow workflows? @pditommaso

CC: @ilveroluca @kikkomep

[ewels]

Thanks for the details @simleo! I think I mentioned this morning that in the https://github.com/nf-core/modules repository for DSL2 tool wrapper modules we have started using pytest-workflow (docs). The GitHub Actions workflow is here - it runs using tool specific config files like this one which in turn run a mini one-process test workflow for that tool.

We do not currently use this for @nf-core pipelines though. At the moment they run on GitHub Actions but their outputs are not verified - we essentially just check the exit status of Nextflow and rely on manually checking the CI logs. We are certainly interested in improving this.

'cc @nf-core/core @KevinMenden @Emiller88

[edmundmiller]

@simleo I've started some progress on this https://github.com/nf-core/rnaseq/pull/546 I've been using them to test some local modules and local subworkflows on a private project, I invited you to have read access on it. I think it's pretty slick so far.

@nf-core/core Still needs to make the decision whether to migrate to pytest-workflows, but if they do I'll be glad to help work out the testing. It's really easy to convert the end2end tests, like I've done with the rnaseq pipeline. I think testing the local modules is also really powerful because it makes running tests locally(or though the CI) much more granular and a quicker feedback loop.

[ewels]

Yes I guess that Planemo is not actually part of Galaxy so it's a similar setup - using a wrapper / helper tool to run tests within Galaxy and validate outputs. So that is what we have been using pytest-workflow to do here so far and it appears to have very similar config YAML files. This tool is nothing to do with Nextflow or nf-core per se though, we're just using it.

I'm not sure that we need direct integration into Nextflow here @simleo unless I'm missing something. Are we answering your question above? I guess if we stick with this then you would then be interested in adding support for pytest-workflow into the ro-crate-py package somehow?

[simleo]

I'm not sure that we need direct integration into Nextflow here

No, that's also not the case for Planemo / Galaxy, as you wrote above.

[simleo]

I've taken a look at pytest-workflow. One problem I see is that it allows arbitrary code execution via the command field, with security implications for the "running tests according to a definition" scenario. It would be very easy for an ro-crate author (either by mistake or maliciously) to insert dangerous code in the test definition. This is different from Planemo, where the YAML file only contains information such as input/output items and assertions.

[simleo]

On the other hand, I guess Nextflow allows arbitrary code execution in the first place, which calls for some form of isolation in the running environment. One difference is that one might expect a lesser need for isolation from the test engine itself. On the plus side, pytest-workflow is general (works for any workflow engine, not just Nextflow) and seems easy to use, so it's something to take into consideration as we move forward.

[ewels]

Yes, I'm not sure that I see much difference in this context. The workflow by definition must be able to run arbitrary code, so I'm not sure that the workflow running the workflow is much of an additional risk.

@simleo - we've been talking with @johanneskoester about collaborating on a set of data that could be used for automated testing: a set of minimal inputs (eg. fastq, bam, vcf files etc). Do you know of anyone already trying to build such a resource?

[simleo]

@simleo - we've been talking with @johanneskoester about collaborating on a set of data that could be used for automated testing: a set of minimal inputs (eg. fastq, bam, vcf files etc). Do you know of anyone already trying to build such a resource?

No. I think most test authors generate ad-hoc inputs or use publicly available ones according to what needs to be tested. Minimal inputs are especially good for functional testing (which is what Life Monitor is mainly about), so a minimal input data set should be pretty useful to test authors, if made publicly available.