search

LUMC / pytest-workflow

Configure workflow/pipeline tests using yaml files.
https://pytest-workflow.readthedocs.io/en/stable/
GNU Affero General Public License v3.0
64 stars 9 forks source link

feature request: Add something like --exclude-tag in addition to --tag #202

Open stianlagstad opened 2 months ago

stianlagstad commented 2 months ago

Hi! Thanks again for this package :100:

I am aware that the -k and -m pytest flags are not supported, for good historical reasons (ref https://github.com/LUMC/pytest-workflow/issues/155). However, would it be possible to add a simpler version of this in pytest-workflow? We have separated our test cases into "slow" and "fast", and we wish to run only the fast ones in pull request builds, and only the slow ones on builds on our main branch. This works, but it requires us to be disciplined about ONLY adding EITHER the tag "slow" OR the tag "fast" to our test cases. We can do that, but it's easy to forget to add a tag, and then the new test case without a tag would never run. What I would instead like to do is to run something like pytest tests --exclude-tag slow on pull request builds, because then we would only need to tag the slow tests with "slow" and we could leave all other tests untagged. This would also help the case where we need more tags.

rhpvorderman commented 2 months ago

Hmm. I think it would be better to rework workflows to run as scope "session" fixtures. Then we can do away with the custom tags and simply use pytests mark expressions that are builtin.

This will require some work though. And it will be hard not to break backwards-compatibility, which I prefer not to do. (allthough it should be possible).

stianlagstad commented 2 months ago

Thanks for responding! I agree that the best solution would be to use newer pytest features and run the workflow commands like that. However, as you say, that's likely a much greater effort. Implementing this smaller feature now could make the tool a bit more useful, and it would not stand in the way of doing "the real fix" later down the line either. With a few pointers, I could look into it and create a PR for the new feature.