search

numirias / pytest-json-report

🗒️ A pytest plugin to report test results as JSON
MIT License
146 stars 39 forks source link

Interoperation with marked tests `@pytest.mark.xyz` #45

Closed atzannes closed 4 years ago

atzannes commented 4 years ago

This is more of a question than an issue.

I have a testsuite and I am using pytest-json-report (which I like very much) to drive our custom CI testing. Our testsuite takes a really long time because of a few long-running (integration) tests. I am marking those tests as "slow" using @pytest.mark.slow, and the idea is to run all the non-slow tests first, and then, if those succeeded, also run the integration tests.

The way I am using pytest-json-report is to get a list of all the test that pytest gets when run with --collect-only and split them into a number of files that are then executed in a distributed way. Now that I'm trying to do this collection phase with the -m 'not slow' flag to select all the fast tests, I am noticing that pytest-json-report does not appear to be marking which tests were deselected by pytest.

I saw in the docs that I could use potentially use the json_metadata fixture to possibly add the slow marker manually, but I would like to avoid this duplication if possible.

Any advice on whether this is possible out-of-the-box, how to hack things together with the current capabilities of the plugin, or how I could modify the plugin to take into account deselected tests during the collection phase will be greatly appreciated.

numirias commented 4 years ago

Good point! While this could be easily done with the provided hooks, I think it makes sense to include that information by default. So, in v1.2.0, there's now an extra deselected boolean for collection results.

Let me know if that works for you.

Just FYI, here is one way it could have been done with hooks (in conftest.py):

deselected = []
def pytest_deselected(items):
    deselected.extend([i.nodeid for i in items])

def pytest_json_modifyreport(json_report):
    json_report['deselected'] = deselected
atzannes commented 4 years ago

Wow, that was speedy turnaround! Thanks! (I was trying my hand at it, but you were much faster) And thanks for the explanation of how to achieve the same with hooks in conftest.py. The whole plugin architecture of pytest making more sense to me after this exercise.