pytest: Test case showed as passed when fixture is failing

[danielbacara]

Hi. First of all thank you for the sphinx-needs and sphinx-test-reports extensions. They are great and helps as to keep everything as code.

I discovered a strange behavior with pytest and sphinx-test-reports when a fixture raises an error the test cases using that fixture are still passing.

Here is a small example, which I hope is reproducible:

# test_example.py
import pytest

@pytest.fixture
def fixture_foo():
    raise Exception('ERROR on fixture')

def test_foo(fixture_foo):
    print("I'm OK")

The above test file is called using the following command:

pytest test_example.py --junitxml=junit.xml

Then in an RST file I simply call the test-report directive:

FOO
=====

.. test-report:: My Report
   :id: FOO
   :file: /workspaces/tmp/junit.xml

It's important to mention that pytest doesn't contain any additional configuration nor sphinx. Also the test suite detects that there is 1 error.

HTML Results Test case is shown as passed

In the imported data, there is 1 error and one test case which is passing.

Packages

pytest 6.2.5
Sphinx 4.4.0 sphinx-data-viewer 0.1.2
sphinx-test-reports 0.3.6
sphinxcontrib-jsmath 1.0.1
sphinxcontrib-needs 0.7.5
sphinxcontrib-plantuml 0.22

[danwos]

The logic in your code to figure out the testcase status is like this:

So if skipped nor failiure is set as an attribute in the testcase, it must be passed.

In the given junit.xml, failure error seems to be used, so we should check our code for both values.

[TruongMinh89]

Hi, No update for this issue? BR, Minh