search

tarpas / pytest-testmon

Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch.
https://testmon.org
MIT License
800 stars 54 forks source link

rerun failures if seemingly nothing changed #161

Closed bencaine-integral closed 3 years ago

bencaine-integral commented 3 years ago

First of all, I love the tool -- it's helped tremendously with our workflow!

The one behavior that is driving us crazy, though, is that testmon caches test failures if there's no update to the files affecting the test. Sometimes, the dependency that fixes the test is outside our python files (e.g. in a library), and fixing that dependency does not fix the test.

Ideally, testmon would run all tests for which dependencies changed or the tests are failing. Would there be interest in making this the default behavior, or at least adding a flag for this behavior?

tarpas commented 3 years ago

In your case you're at least able to "fix" the testmon data with --testmon --lf where "--lf" is the standard pytest option.

As for default, I decided for the behaviour we have because I thought it's more correct and more predictable and therefore better. But given your feedback I'll think about it more. If you think about getting your hands dirty with the code contact me at tibor@testmon.org .

bencaine-integral commented 3 years ago

--lf works! Thank you!

tarpas commented 3 years ago

I'll actually close this because I'm tracking this idea in #148 . Thanks for feedback!

bencaine-integral commented 3 years ago

Of course!

I just realized that --lf also re-runs all the tests (even the ones that succeeded), so it's a "fix" if we suspect a cached failure, but it'd be great to have a way to always re-run failed tests and not ones that succeeded.

blueyed commented 3 years ago

I just realized that --lf also re-runs all the tests (even the ones that succeeded)

That should not be the case really and would be another bug then - either in pytest-testmon or pytest. IIRC this might be something planned to address correctly in testmon (similar to -k is handled / bypasses testmon currently/maybe). Sorry for not having checked it properly, but in general --lf is meant to only run "last failures".