Open adamrtalbot opened 3 weeks ago
I like that idea very much.
I've been working on https://github.com/nf-core/rnaseq/pull/1335
In which I checked/fixed/implemented stub and stub tests for lots of modules and subworkflows.
And I found it was quite cumbersome to just rewrite the test with stub.
For all of these what I was doing was just copy-paste the regular test, adding - stub
in the name, adding options "-stub"
, and snapshotting process.out
or workflow.out
.
So that feels like a lot of code duplication for not much.
Thanks, that definitely makes sense to save time and avoid duplicated code. I am currently busy with teaching (end of semester) and trying to release 0.9.0 really soon. But afterwards, I'll think about how we could implement Adam's ideas.
What if there were just macros?
https://github.com/avajs/ava/blob/main/docs/01-writing-tests.md#reusing-test-logic-through-macros
Or this might be some wizardry but what about Data tables support?
https://spockframework.org/spock/docs/2.3/data_driven_testing.html#data-tables
It would also be nice if an option would exist that allows the user to specify which tests should be run in stub
and normal
mode. This way we can start testing all tests in stub mode very easily without having to rewrite a new test for each test variation. (I'm thinking something along the lines of a new option called testMode
that could take a list of test modes => there's probably a better way to do this, but I'm just spitballing here :grin: )
What if there were just macros?
https://github.com/avajs/ava/blob/main/docs/01-writing-tests.md#reusing-test-logic-through-macros
Or this might be some wizardry but what about Data tables support?
https://spockframework.org/spock/docs/2.3/data_driven_testing.html#data-tables
These are both nice but I think they're a good way of testing a matrix of inputs rather than for stubs. I think stubs should be baked into the normal tests, so a Spock equivalent might be:
when:
include { process_a } from 'process_a'
then:
expect:
process_a(INPUT) == EXPECTED
where:
INPUT | EXPECTED
1 | 1
and:
2 | 2
4 | 2
stub:
process_a.success
process_a.out.output[0].name == "touch.txt"
This is basically writing a separate set of assertions into the main test for stubs.
It would also be nice if an option would exist that allows the user to specify which tests should be run in stub and normal mode. This way we can start testing all tests in stub mode very easily without having to rewrite a new test for each test variation. (I'm thinking something along the lines of a new option called testMode that could take a list of test modes => there's probably a better way to do this, but I'm just spitballing here 😁 )
I think we're agreed - if we can, we should write stubs into the main body of the test rather than as a separate test. I guess the question is how.
Writing a separate stub body sounds like a good approach (especially it is similar to nextflow itself). When a stub body is present, we start the test with the stub flag first and check only the assertions within the stub body. Next, we can run it in normal mode, checking the regular assertions.
Based on this, we can implement different strategies, such as skipping the regular test if the stub fails, etc.
Currently, we have to write an additional test with stub enabled with separate assertions and parameters to use stubs. However, we would generally like to run nf-test with the stubs as close to the real thing as possible and a separate test creates a divergence between the main test and the stub test. Ideally, we would run the stub test prior to the 'real' test to make sure it makes sense before spending time on the main test.
Instead of writing a separate test, nf-test could have a native stub mode which allows us to reuse the same test but with lower stringency. This would enable us to run nf-test in a 'pre-run' mode and use the same code and logic as the main nf-test.
Suggested implemented:
nf-test test -stub
which runs Nextflow in stub mode-stub
.