search

tarpas / pytest-testmon

Selects tests affected by changed files. Executes the right tests first. Continuous test runner when used with pytest-watch.
https://testmon.org
MIT License
800 stars 54 forks source link

Feature request: rerun all tests if dependencies change #159

Closed michamos closed 3 years ago

michamos commented 3 years ago

First let me thank you for this great project. It works very well and is a huge productivity boost when having a large and slow testsuite.

One small issue I have is dependency management. When external dependencies change, it would be good to re-run all tests, as they are not tracked. Would it be possible to supplement pytest-testmon with a feature allowing it to detect if dependencies have changed, and if so re-run all tests?

Edit: after some investigation, this shouldn't be too hard by using pkg_resources and storing a normalized output of pkg_resources.working_set or even its hash (adapted from https://stackoverflow.com/a/65040585):

In [1]: import pkg_resources                                                                                         

In [2]: "\n".join(sorted(str(p) for p in pkg_resources.working_set))                                                 
Out[2]: 'Pillow 8.0.1\nPygments 2.7.3\nbackcall 0.2.0\nbackoff 1.10.0\ncertifi 2020.12.5\nchardet 3.0.4\ncycler 0.10.0\ndecorator 4.4.2\nidna 2.10\nipdb 0.13.4\nipython 7.16.1\nipython-genutils 0.2.0\njedi 0.17.2\nkiwisolver 1.3.1\nmatplotlib 3.3.3\nnumpy 1.19.4\norjson 3.4.6\nparso 0.7.1\npexpect 4.8.0\npickleshare 0.7.5\npip 20.2.4\nprompt-toolkit 3.0.3\nptyprocess 0.6.0\npyparsing 2.4.7\npython-dateutil 2.8.1\npython-rapidjson 0.9.4\nrequests 2.25.0\nsetuptools 50.3.2\nsimplejson 3.17.2\nsix 1.15.0\ntqdm 4.54.1\ntraitlets 4.3.3\nujson 4.0.1\nurllib3 1.26.2\nwcwidth 0.2.5\nwheel 0.35.1'
tarpas commented 3 years ago

I've been playing with this idea, but didn't know about pkg_resources.working_set. Very good suggestions, thanks!

tarpas commented 3 years ago

All: how usable is this if there is no warning why everything is re-executed? @michamos

michamos commented 3 years ago

Couldn't you add a message in the plugin info, where you currently display the status of test collection, selection, etc.?

tarpas commented 3 years ago

Sure, I'll work on that.