search

LUMC / pytest-workflow

Configure workflow/pipeline tests using yaml files.
https://pytest-workflow.readthedocs.io/en/stable/
GNU Affero General Public License v3.0
64 stars 9 forks source link

Rerun last failed workflows in same tmp directory? #174

Open glormph opened 1 year ago

glormph commented 1 year ago

Hi, I am trying to apply pytest-workflow to my nextflow pipeline. NF itself comes with a -resume option, which makes it skip existing, previously successful workflow step. Since the pipeline itself has some slow steps, this option is great when testing, even with more minimal data. I have however not figured out how to use this option in pytest-workflow. I tried running pytest --kwdof --color=yes --lf, but it creates a new tmpdir every time I rerun.

I'm not sure if I just haven't figured out how to do it, if it could be a nice feature, or that it shouldn't be implemented :)

rhpvorderman commented 1 year ago

I think that will be quite hard to implement. Not too mention that various workflow engines have different resume requirements, and the project should remain workflow engine agnostic as that is its greatest strength.

In this case I think you are better of running the command directly on the terminal instead. Keep debugging until it can run from start to finish and then use pytest workflow to add tests.

As for the slow steps, you could try using only a subset of the data as test data set. That might speed things up and should be good enough for testing with pytest-workflow.

Does this help you?

glormph commented 1 year ago

Thanks (that was fast!) Ah yes, I understand.

As for subsetting, yes, I've cut my data ("normal" data would take ~1 hour to run), and a 5 minute wait is fine when it's a first time run, or on github actions on a PR, especially when it is successful. I'll use the direct pipeline command locally for when quicker feedback is needed then.

yfarjoun commented 1 year ago

a Similar, but not-quite-the-same request would be to rerun the tests but not the workflow since, as the OP stated, that is often the time-consuming part... and when adjusting the yml file I find that iterating on that would be sped up significantly if one could run pytest .... --workdir <old tmp dir> --do_not_rerun_commands ... or something like that.

rhpvorderman commented 1 year ago

@yfarjoun, I don't understand that bit. When your workflow fails tests it is incorrect, so it needs changing. When it has changed, it needs to run again to see the changes in effect. What is the purpose of running the tests again if the workflow is not changed?

yfarjoun commented 1 year ago

thanks for responding so promptly!

I want to be able to modify the yml in the files section and not change the workflow at all. for example, add/modify an md5, add values to a "contains" command. etc. I do not need to re-run the workflow itself for that.

rhpvorderman commented 1 year ago

Ah, I see. So this is more of a test development feature. Can you make a separate issue for that? I can't make any promises on when (and if) I will get to it, but at least it will be more visible in case somebody else wants to contribute. Plus, if we get a range of issues on improvements it is a nice internship project. (We had one pytest-workflow internship and we were very happy with how that turned out.)