search

zopefoundation / zope.testrunner

This package provides a flexible test runner with layer support.
https://zopetestrunner.readthedocs.io/
Other
2 stars 23 forks source link

Repeated execution of the same test #108

Open d-maurer opened 4 years ago

d-maurer commented 4 years ago

In some cases, the same test is executed repeatedly.

I have observed this behavior with the following scenario: (test) module ma with a TestCase class A; (test) module "mb" contains from ma import A and derives TestCase class B from A. In my case, the motivation has been to run all A tests in a different set up (defined by B). When I run the test suite, all A tests are executed twice; in addition, the B tests are executed (as intented).

The explanation is quite simple: when the runner collects the TestCase classes from a given module, it does not check whether the class was actually defined by the module; it is sufficient that it contains a reference to the class -- this may be a feature. Therefore, in the scenario above, mb gives test classes A and B; A is selected a second time from ma.

When one is aware of this behavior one can fairly easily avoid repetitions: replace from ma import A ... class B(A) ... by import ma ... class B(ma.A) .... But many people won't be aware. Therefore, I suggest to check for repetitive execution of the same test is the identical context (--> layer) and avoid it.

jamadden commented 4 years ago

The stdlib unittest runner does the same thing in Python 2 and Python 3. Given module-level fixtures (setUpModule), that definitely makes sense.

I'm not sure zope.testrunner supports those (if not I think it should, same with setUpClass), but there are probably other reasons for it.

Since I don't see a clear win, this mostly seems like a gratuitous difference from the standard library, so for right now I would be -0.5.

d-maurer commented 4 years ago

Jason Madden wrote at 2020-7-7 03:30 -0700:

The stdlib unittest runner does the same thing in Python 2 and Python 3. Given module-level fixtures (setUpModule), that definitely makes sense.

I'm not sure zope.testrunner supports those (if not I think it should, same with setUpClass), but there are probably other reasons for it.

I do not think it does. But, if it does, it knows this (because it must apply it) and can then allow repeated test execution when the different occurrence may indeed be different (due to different setUp/tearDown).