when the testcase fails at the fixture level, pytest_testrail.plugin does not publish results to testrail

[Sushma-Kattinti-Bose]

@allankilpatrick

Problem statement - When the testcase fails at the fixture level, pytest_testrail.plugin does not publish results to testrail - it says untested

• We are using pytest_testrail.plugin to push results to testrail which works perfectly when the test passes/fails
• When the pytest contains a fixture and the test fails at the fixture level then the results are not published to testrail

Could you please guide me on how I should proceed with it. Please let me know if you need more information

I am sorry, I did not know how to reach/contact the author of this repository. Appreciate your help in advance

Thanks

[allankp]

@Sushma-Kattinti-Bose Can you give me some example code please?

I think I understand the issue, could be that when a test fails on a fixture pytest_runtest_makereport is not hit or the test is not in a call status

[Sushma-Kattinti-Bose]

It is not in call status when we are executing fixtures. It is in setup status

[Sushma-Kattinti-Bose]

Following test will fail at fixture level. Results are not pushed to testrail - Testrail interface says "untested"

conftest.py

from pytest_testrail.plugin import pytestrail

@pytest.hookimpl
def pytest_collection_modifyitems(config, items):
    """
    :param config: PyTest command line argument
    :param items: List of collected test items
    """
    if config.getoption('--testrail'):
        tests = retrieve_test_cases() # Returns test case id and test case title in a list format
        for item in items:
            for test in tests:
                for testid, testname in test.iteritems():
                    if item.name == testname:
                        item.add_marker(pytestrail.case(testid))

@pytest.fixture()
def setup():
    assert False

test_example.py

@pytest.mark.usefixtures('setup')
def test_example():
    print "PASS"

Command line

pytest -sv test_example.py \
                           --testrail \
                           --tr-run-id=123456 \
                           --tr-config=../../../testrailsettings.cfg 

[allankp]

What happens if you change test_example.py to:

def test_example(setup):
    print "PASS"

[Sushma-Kattinti-Bose]

I see the same issue -It will not as it did not come into call state at that point

[allankp]

I think modifying : https://github.com/allankp/pytest-testrail/blob/master/pytest_testrail/plugin.py#L188 to if rep.when in ['call', 'setup'] and testcaseids: Would result in duplicated test cases in the run, which isn't good.

Technically showing as 'untested' I think is correct, as the test itself was never actually run. It was the setup that failed.

[Sushma-Kattinti-Bose]

Right, the test itself never actually ran as it failed at the fixture level. You have any alternative way to push results to Testrail when it fails at the fixture level?

[allankp]

I feel this is working as intended and the result of untested is correct. Therefore the result being shown in TestRail is also correct. If you would like to propose another solution, I'd be happy to take a look.

[Sushma-Kattinti-Bose]

Sure, thanks!

[allankp]

Looking forward to seeing the PR, closing this off.