search

pytest-dev / pytest-rerunfailures

a pytest plugin that re-runs failed tests up to -n times to eliminate flakey failures
Other
390 stars 84 forks source link

VS Code compatibility help #280

Open eleanorjboyd opened 2 weeks ago

eleanorjboyd commented 2 weeks ago

Hi! I am the maintainer of testing in the Python extension for VS Code. I was doing a review of top pytest extensions to check their compatibility with the extension. I tried pytest-rerunfailures and found it didn't work correctly in the case that the test failed, then later passed. Instead of showing the test as passing, it only took the first failure and displayed that in the UI. I am looking for advice on how to make sure the plugin works correctly- below I explain a bit about how the extension works and hopefully that can give you enough context to provide some advice. Thanks!

Pytest in the Python extension works with a custom pytest plugin for VS Code. In this plugin it collects test results after each test runs and sends a payload back to the main extension host to display to the user (this allows for dynamic run, where the results come in as they finish instead of all at the end of the entire run). The extension also sends back payloads on exceptions, which includes the def pytest_exception_interact(node, call, report): hook from pytest.

What I am noticing as I am testing with the pytest-rerunfailures plugin is that a test which fails once, but still can rerun, calls the hook def pytest_exception_interact(node, call, report):. From my investigation it seems that the node.execution_count variable keeps track of how many times that test has run. Then if the rerun value is set via a pytest marker, the max attempts can be found in the marker kwargs or if the reruns are defined from the command line argument it is stored on the _rerun attribute on report.

The best solution I can determine right nwo is to not send the failure to the extension if the def pytest_exception_interact(node, call, report): is called when the node.execution_count < max_attempts, but the max attempts can be defined a few different ways. Is this the correct way to handle this situation and the best way to check to see if the given test has more reruns to execute?

Sorry for the long explanation and please let me know what other information I can provide for clarity. If you are not familiar, you can try testing in VS Code by clicking on the "beaker" icon in the sidebar of VS Code and from there you can setup testing with pytest. I appreciate your help as I try and get the best experience for this plugin in VS Code! Thanks

The-Compiler commented 2 weeks ago

While I don't have an answer, with my pytest maintainer hat on, I wanted to say that I really appreciate how you're working with the community and ecosystem around pytest, in order to make the integration into VS Code better for everyone!

eleanorjboyd commented 2 weeks ago

@The-Compiler Thank you so much, that is very kind! I have appreciated working with you on pytest in the past and love the whole pytest community! Speaking of plugins, if you find any you like that don't work in VS Code, feel free to reach out. Thanks!