pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.94k stars 2.66k forks source link

Inverse versions of `python_files` and `python_functions`. #10237

Open Asday opened 2 years ago

Asday commented 2 years ago

Suppose my tests are in a well-marked file such as test_queue.py. It doesn't make sense to name tests in the form test_queue_starts_empty(), test_queue_size_increases_after_push(), and so forth, because I know they're tests. I can set python_functions = * to drop the test_ prefix, which is nice until I end up with a passing fancy to extract something out of a test to a utility function without turning it into a fixture, then suddenly my utility function is run as a test.

That's not the worst thing ever, because if my util function doesn't do any asserting it won't cause a CI failure or what have you, but it still counts as +1 test, which isn't ideal.

Were I able to instead set python_ignore_functions = _*, that would be lovely.

The exact same argument applies to python_files - let's say all of my tests are under tests/ - they have no need to be named test_queue.py, test_stack.py, etc., because I know they're test modules by dint of being in the directory where tests go. Again, python_files = * is great, until I end up with a util function I want to reuse across multiple files (yet somehow still not wanting to turn it into a fixture), and now it's being picked up as a test module.

nicoddemus commented 2 years ago

Hi @Asday,

Thanks for the write up!

Being explicit about test functions also has advantages: it is convenient to be able to search for test_ to find test functions, new contributors to a project immediately recognize def test_ as test functions, etc. For this reason I'm -1 on the idea, feels like feature bloat to the core.

But I'm sure it should be possible to implement that as a separate plugin, if you feel strongly about this feature.

Asday commented 2 years ago

I certainly agree that convention is a great thing, but the fact that it's configurable at all means that there's precedent for it being configurable. The docs give the example of changing "test" for "check" in conventions with no opinionation on whether that convention is better or worse.

From that, I would propose that the expected workflow for new contributors to learn the conventions for a project is for them to read the configuration file(s), much the same way that they'd be expected to read .editorconfig were they choosing to use an editor without support for it, to run go fmt before committing, or use K&R/1TBS based on surrounding code. Admittedly the go fmt example isn't the best because that community is highly opinionated on what is the correct convention, but I'm sure you'll forgive me.

Put simply - I think the existence of the current options sets precedence for their inverse.

RonnyPfannschmidt commented 2 years ago

I reject the proposal based on the notion that it adds too many footguns

There has to be a ballance between enabling behaviour and feature creep

In particular the existence of certain options does never have to imply the existence of inverse

Asday commented 2 years ago

That's very fair. Could I ask for a little elaboration on what sort of footguns python_ignore_functions would introduce?

Asday commented 2 years ago

Oh hold on I'm a dumbass this already exists kinda.

collect_ignore_glob = ["*/factories.py"] in conftest.py, which is documented here.

Not super pleased that it's separate from the rest of my pytest config in an executable file, but I am super pleased that I can do it without resorting to writing some sort of plugin or forking pytest.