search

fundakol / pytest-jira-xray

Plugin for pytest to upload test results to JIRA XRAY
Apache License 2.0
26 stars 18 forks source link

Reporting duplicated IDs but not duplicated IDs in the tests #80

Closed RoCCoCo13 closed 1 year ago

RoCCoCo13 commented 1 year ago

Hi there,

I am trying to report the test results to a list of test cases in a parametrized test, but when I try to execute the tests it reports that there are duplicated IDs of the tests.

I have checked the list and I couldn't find any duplicated ID.

Test:

class TestImportedToDeduplication:
    status_filter = {"status": "user.status LIKE 'DUP_CHECKED' or user.status LIKE 'DUPLICATED'"}

    @pytest.mark.xray(
        [
            "VSDS-14750",
            "VSDS-14752",
            "VSDS-14753",
            "VSDS-14771",
            "VSDS-14772",
            "VSDS-14773",
            "VSDS-14774",
            "VSDS-14775",
            "VSDS-14776",
            "VSDS-14777",
            "VSDS-14778",
            "VSDS-14779",
            "VSDS-14780",
            "VSDS-14781",
            "VSDS-14782",
            "VSDS-14783",
            "VSDS-14784",
            "VSDS-14785",
            "VSDS-14786",
            "VSDS-14787",
            "VSDS-14788",
            "VSDS-14789",
            "VSDS-14790",
            "VSDS-14791",
            "VSDS-14792",
            "VSDS-14793",
            "VSDS-14794",
            "VSDS-14795",
            "VSDS-14796",
            "VSDS-14797",
            "VSDS-14798",
            "VSDS-14799",
            "VSDS-14800",
            "VSDS-14801",
            "VSDS-14802",
            "VSDS-14803",
            "VSDS-14804",
            "VSDS-14805",
            "VSDS-14806",
            "VSDS-14807",
            "VSDS-14808",
            "VSDS-14809",
            "VSDS-14810",
            "VSDS-14811",
            "VSDS-14812",
            "VSDS-14813",
            "VSDS-14814",
            "VSDS-14815",
            "VSDS-14816",
            "VSDS-14817",
            "VSDS-14818",
            "VSDS-14819",
            "VSDS-14820",
            "VSDS-14821",
            "VSDS-14822",
            "VSDS-14823",
            "VSDS-14824",
            "VSDS-14825",
            "VSDS-14826",
            "VSDS-14827",
            "VSDS-14828",
            "VSDS-14829",
            "VSDS-14830",
            "VSDS-14831",
            "VSDS-14832",
            "VSDS-14833",
            "VSDS-14834",
            "VSDS-14835",
            "VSDS-14836",
            "VSDS-14837",
            "VSDS-14838",
            "VSDS-14839",
            "VSDS-14840",
            "VSDS-14841",
            "VSDS-14842",
            "VSDS-14843",
            "VSDS-14844",
            "VSDS-14845",
            "VSDS-14846",
            "VSDS-14847",
            "VSDS-14848",
            "VSDS-14849",
            "VSDS-14850",
            "VSDS-14851",
            "VSDS-14852",
            "VSDS-14853",
            "VSDS-14854",
            "VSDS-14855",
            "VSDS-14856",
            "VSDS-14857",
            "VSDS-14858",
            "VSDS-14859",
            "VSDS-14860",
            "VSDS-14861",
            "VSDS-14862",
            "VSDS-14863",
            "VSDS-14864",
            "VSDS-14865",
            "VSDS-14866",
            "VSDS-14867",
            "VSDS-14868",
            "VSDS-14869",
            "VSDS-14870",
            "VSDS-14871",
            "VSDS-14872",
            "VSDS-14873",
            "VSDS-14874",
            "VSDS-14875",
            "VSDS-14876",
            "VSDS-14877",
            "VSDS-14878",
            "VSDS-14879",
            "VSDS-14880",
            "VSDS-14881",
            "VSDS-14882",
            "VSDS-14883",
            "VSDS-14884",
            "VSDS-14885",
            "VSDS-14886",
            "VSDS-14887",
            "VSDS-14888",
            "VSDS-14889",
            "VSDS-14890",
            "VSDS-14891",
            "VSDS-14892",
            "VSDS-14893",
            "VSDS-14894",
            "VSDS-14895",
        ]
    )
    @pytest.mark.parametrize("credit_reference_id", [False, True])
    @pytest.mark.parametrize("session_start", [False, True])
    @pytest.mark.parametrize("rfid", [False, True])
    @pytest.mark.parametrize("emaid", [False, True])
    @pytest.mark.parametrize("evcoid", [False, True])
    @pytest.mark.parametrize("evseid", [False, True])
    @pytest.mark.parametrize("external_cdr_id", [False, True])
    @pytest.mark.parametrize("getCDR_fixt", [["NEW"]], indirect=["getCDR_fixt"])
    @pytest.mark.parametrize(
        "setup_asb_and_sub_management",
        [[status_filter]],
        indirect=["setup_asb_and_sub_management"],
    )
    def test_NEW_to_DUP_CHECKED(
        self,
        setup_asb_and_sub_management,
        getCDR_fixt,
        copy_cdr_with_updated_fields,
        external_cdr_id,
        evseid,
        evcoid,
        rfid,
        emaid,
        session_start,
        credit_reference_id,
    ):
...
test-content
...

Versions

$ python --version
Python 3.10.0
$ pip freeze | grep pytest
pytest==7.2.0
pytest-cov==3.0.0
pytest-html==3.1.1
pytest-jira-xray==0.8.8
pytest-metadata==2.0.1
pytest-xdist==3.3.1

Logs:

$ pytest -c tests_data/deroem/dev/pytest.ini --rootdir=tests_middleware/ tests_middleware/mqs_updates_cdr/test_cdr_mw_dedup.py --jira-xray --api-key-auth --testplan VSDS-14487
============================================================================================================ test session starts ============================================================================================================ platform win32 -- Python 3.10.0, pytest-7.2.0, pluggy-1.0.0 -- C:\Users\alejandro.mendez\AppData\Local\Programs\Python\Python310\python.exe
cachedir: .pytest_cache
metadata: {'Python': '3.10.0', 'Platform': 'Windows-10-10.0.22000-SP0', 'Packages': {'pytest': '7.2.0', 'py': '1.11.0', 'pluggy': '1.0.0'}, 'Plugins': {'cov': '3.0.0', 'html': '3.1.1', 'metadata': '2.0.1', 'jira-xray': '0.8.8', 'xdist': '3.3.1', 'tavern': '1.25.2'}, 'JAVA_HOME': 'C:\\Program Files\\Java\\jdk-11.0.10'}
rootdir: C:\Users\alejandro.mendez\workspace\middleware-testing\tests_middleware, configfile: ..\tests_data\deroem\dev\pytest.ini
plugins: cov-3.0.0, html-3.1.1, metadata-2.0.1, jira-xray-0.8.8, xdist-3.3.1, tavern-1.25.2
collected 128 items
INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Local\Programs\Python\Python310\lib\site-packages\_pytest\main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Local\Programs\Python\Python310\lib\site-packages\_pytest\main.py", line 323, in _main
INTERNALERROR>     config.hook.pytest_collection(session=session)
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Local\Programs\Python\Python310\lib\site-packages\_pytest\main.py", line 334, in pytest_collection
INTERNALERROR>     session.perform_collect()
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Local\Programs\Python\Python310\lib\site-packages\_pytest\main.py", line 667, in perform_collect
INTERNALERROR>     hook.pytest_collection_modifyitems(
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_hooks.py", line 265, in __call__
INTERNALERROR>     return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_manager.py", line 80, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_callers.py", line 60, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_result.py", line 60, in get_result
INTERNALERROR>     raise ex[1].with_traceback(ex[2])
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Roaming\Python\Python310\site-packages\pluggy\_callers.py", line 39, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Local\Programs\Python\Python310\lib\site-packages\pytest_xray\xray_plugin.py", line 169, in pytest_collection_modifyitems
INTERNALERROR>     self._verify_jira_ids_for_items(items)
INTERNALERROR>   File "C:\Users\alejandro.mendez\AppData\Local\Programs\Python\Python310\lib\site-packages\pytest_xray\xray_plugin.py", line 93, in _verify_jira_ids_for_items
INTERNALERROR>     raise XrayError(f'Duplicated test case ids: {duplicated_jira_ids}')
INTERNALERROR> pytest_xray.exceptions.XrayError: Duplicated test case ids: ['VSDS-14750', 'VSDS-14752', 'VSDS-14753', 'VSDS-14771', 'VSDS-14772', 'VSDS-14773', 'VSDS-14774', 'VSDS-14775', 'VSDS-14776', 'VSDS-14777', 'VSDS-14778', 'VSDS-14779', 'VSDS-14780', 'VSDS-14781', 'VSDS-14782', 'VSDS-14783', 'VSDS-14784', 'VSDS-14785', 'VSDS-14786', 'VSDS-14787', 'VSDS-14788', 'VSDS-14789', 'VSDS-14790', 'VSDS-14791', 'VSDS-14792', 'VSDS-14793', 'VSDS-14794', 'VSDS-14795', 'VSDS-14796', 'VSDS-14797', 'VSDS-14798', 'VSDS-14799', 'VSDS-14800', 'VSDS-14801', 'VSDS-14802', 'VSDS-14803', 'VSDS-14804', 'VSDS-14805', 'VSDS-14806', 'VSDS-14807', 'VSDS-14808', 'VSDS-14809', 'VSDS-14810', 'VSDS-14811', 'VSDS-14812', 'VSDS-14813', 'VSDS-14814', 'VSDS-14815', 'VSDS-14816', 'VSDS-14817', 'VSDS-14818', 'VSDS-14819', 'VSDS-14820', 'VSDS-14821', 'VSDS-14822', 'VSDS-14823', 'VSDS-14824', 'VSDS-14825', 'VSDS-14826', 'VSDS-14827', 'VSDS-14828', 'VSDS-14829', 'VSDS-14830', 'VSDS-14831', 'VSDS-14832', 'VSDS-14833', 'VSDS-14834', 'VSDS-14835', 'VSDS-14836', 'VSDS-14837', 'VSDS-14838', 'VSDS-14839', 'VSDS-14840', 'VSDS-14841', 'VSDS-14842', 'VSDS-14843', 'VSDS-14844', 'VSDS-14845', 'VSDS-14846', 'VSDS-14847', 'VSDS-14848', 'VSDS-14849', 'VSDS-14850', 'VSDS-14851', 'VSDS-14852', 'VSDS-14853', 'VSDS-14854', 'VSDS-14855', 'VSDS-14856', 'VSDS-14857', 'VSDS-14858', 'VSDS-14859', 'VSDS-14860', 'VSDS-14861', 'VSDS-14862', 'VSDS-14863', 'VSDS-14864', 'VSDS-14865', 'VSDS-14866', 'VSDS-14867', 'VSDS-14868', 'VSDS-14869', 'VSDS-14870', 'VSDS-14871', 'VSDS-14872', 'VSDS-14873', 'VSDS-14874', 'VSDS-14875', 'VSDS-14876', 'VSDS-14877', 'VSDS-14878', 'VSDS-14879', 'VSDS-14880', 'VSDS-14881', 'VSDS-14882', 'VSDS-14883', 'VSDS-14884', 'VSDS-14885', 'VSDS-14886', 'VSDS-14887', 'VSDS-14888', 'VSDS-14889', 'VSDS-14890', 'VSDS-14891', 'VSDS-14892', 'VSDS-14893', 'VSDS-14894', 'VSDS-14895']

Any idea about the problem?

Thanks!

fundakol commented 1 year ago

pytest.mark.parametrize generates multiple tests with the same Xray ids from @pytest.mark.xray decorator. This is why you see the exception. You can try to run pytest with the option: --allow-duplicate-ids. It allows to have many tests with same Xray ID, but the overall status for the Xray test is calculated from all tests with the same ID.

Some example:

@pytest.mark.xray('Jira-1')
def test_foo_1():
    assert True

@pytest.mark.xray('Jira-1')
def test_foo_2():
    assert False

This will generate report with one XRAY test which has status FAIL because one test passed and one failed. 

{
 "testKey": "Jira-1",
 "status": "FAIL",
 "comment": "..." 
}

This is similar to your case with parametrize.

RoCCoCo13 commented 1 year ago

Hi there,

So If we consider the outcome of a parametrized test as results for more than one test case in jira/xray, we should always use --allow-duplicate-id to make it work, right?

In my opinion, this is a quite common case, do you think that it would worth mentioning it in the readme?

Many thanks for your work @fundakol!

Cheers