search

LUMC / pytest-workflow

Configure workflow/pipeline tests using yaml files.
https://pytest-workflow.readthedocs.io/en/stable/
GNU Affero General Public License v3.0
63 stars 9 forks source link

Optionally Re-run the tests but not the workflow to help develop tests #176

Open yfarjoun opened 1 year ago

yfarjoun commented 1 year ago

Slightly different from https://github.com/LUMC/pytest-workflow/issues/174 (not a duplicate)

Once I have a workflow and code set up, I iterate on the tests that I want to add, adding md5sum contains elements to the yml file, but not actually changing the workflow or the code behind it.

It would be helpful if there was a way to just run the "lower part" of each workflow test.

In the envisioned implementation the user would

This would cause pytest workflow to use the old working directory, not delete file in it, skip the first step of each test (running the workflow) and evaluate the rest of the tests as if that step succeeded.

rhpvorderman commented 1 year ago

I think this can be collapsed in a single command line flag that provides the old TMP folder where all the workflows are run. This then also disables launching any workflows and simply the workflow directory provided is used to run all the tests again. I think this is doable to implement without too much code.

So now we just need a nice flag. I am thinking of --reuse-workflow-dir <WORKFLOW_DIR>. @yfarjoun, @DavyCats , @Redmar-van-den-Berg any suggestions or feedback?

Redmar-van-den-Berg commented 1 year ago

I think this would be a nice addition, especially for slow tests where you run your entire pipeline with some fake data, which can take a few minutes each time.

wholtz commented 12 months ago

This is functionality that I would also find useful.

As for the name of the flag, --reuse-workflow-dir <WORKFLOW_DIR> could be interpreted as simply reusing the the directory as the place to execute the workflow. I will suggest --reevaluate-workflow-dir <WORKFLOW_DIR> as it is more descriptive of the action being performed.

rhpvorderman commented 12 months ago

@wholtz Excellent suggestion. I agree. That is a nice descriptive flag.

stianlagstad commented 4 months ago

I would very much like a flag like --reevaluate-workflow-dir <WORKFLOW_DIR>, to allow quicker iterations of the tests (not the workflow). Any initial ideas on how to implement this, to get me started?