reportportal / agent-python-pytest

Framework integration with PyTest
Apache License 2.0
94 stars 103 forks source link

Fixed mtultiple chained errors thrown when using pytest Tavern #367

Closed virdok closed 2 months ago

virdok commented 2 months ago

Description

A framework exists above Pytest which is called Tavern. It is using YAML files to execute API tests. During the execution of the tests in this framework and due to the difference it has in the structure of the test suites in respect to BDD for example, some changes were needed as some exceptions were thrown and the integration of the pytest execution with Reportportal was not possible / blocked.

Points of failure

Original name attribute

The first point of failure is due to the fact that the Tavern framework do not use the originalname in the test item. This change fixes the issue with the originalname checking if its existing in the item as an attribute.

If the originalname attribute does not exist then instead of using the item.name defined in this if / else statement, it was throwing the following error:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 349, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 88, in _multicall
INTERNALERROR>     next(wrapper_gen)  # first yield
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/plugin.py", line 263, in pytest_runtest_protocol
INTERNALERROR>     service.start_pytest_item(item)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 129, in wrap
INTERNALERROR>     func(*args, **kwargs)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 710, in start_pytest_item
INTERNALERROR>     self._process_metadata_item_start(current_leaf)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 650, in _process_metadata_item_start
INTERNALERROR>     leaf['code_ref'] = self._get_code_ref(item)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 460, in _get_code_ref
INTERNALERROR>     item.originalname is not None \
INTERNALERROR> AttributeError: 'YamlItem' object has no attribute 'originalname'

Parent attribute

In the line 467 a check for the parent attribute has been added. The reason of this is the item structure of the Tavern framework in comparison with the rest of the Pytest Item structure. After getting the first time the parent object in line 461, then the Tavern framework does not have at all parents set. (Although it depends somehow to the test implementation, but always after the root object, parent is NoneType).

Falling in this case the framework throws the following exception:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 349, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 88, in _multicall
INTERNALERROR>     next(wrapper_gen)  # first yield
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/plugin.py", line 263, in pytest_runtest_protocol
INTERNALERROR>     service.start_pytest_item(item)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 129, in wrap
INTERNALERROR>     func(*args, **kwargs)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 709, in start_pytest_item
INTERNALERROR>     self._process_metadata_item_start(current_leaf)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 649, in _process_metadata_item_start
INTERNALERROR>     leaf['code_ref'] = self._get_code_ref(item)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 466, in _get_code_ref
INTERNALERROR>     classes.append(parent.name)
INTERNALERROR> AttributeError: 'NoneType' object has no attribute 'name'

So in the line 467 we check if parent attribute exists so as to step into it otherwise breaks the loop. If the loop is not broken at this point it ends up having an infinite loop.

Parent name

In the line 464 we check if the attribute name exists in the parent. The reason of doing so is the fact that at a point the parent object is not related to the test (It may be Session object for example) and this was throwing a name does not exist error as follows:

INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 270, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 324, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/_pytest/main.py", line 349, in pytest_runtestloop
INTERNALERROR>     item.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 182, in _multicall
INTERNALERROR>     return outcome.get_result()
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_result.py", line 100, in get_result
INTERNALERROR>     raise exc.with_traceback(exc.__traceback__)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pluggy/_callers.py", line 88, in _multicall
INTERNALERROR>     next(wrapper_gen)  # first yield
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/plugin.py", line 263, in pytest_runtest_protocol
INTERNALERROR>     service.start_pytest_item(item)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 129, in wrap
INTERNALERROR>     func(*args, **kwargs)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 708, in start_pytest_item
INTERNALERROR>     self._process_metadata_item_start(current_leaf)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 648, in _process_metadata_item_start
INTERNALERROR>     leaf['code_ref'] = self._get_code_ref(item)
INTERNALERROR>   File "/Users/dkirchan/Library/Caches/pypoetry/virtualenvs/qa-earnings-app-qps9_NMR-py3.9/lib/python3.9/site-packages/pytest_reportportal/service.py", line 467, in _get_code_ref
INTERNALERROR>     parent = parent.parent
INTERNALERROR> AttributeError: 'NoneType' object has no attribute 'parent'
codecov-commenter commented 2 months ago

Codecov Report

Attention: Patch coverage is 80.00000% with 1 line in your changes missing coverage. Please review.

Project coverage is 70.66%. Comparing base (fbaa07e) to head (be082f8). Report is 5 commits behind head on develop.

Files Patch % Lines
pytest_reportportal/service.py 80.00% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## develop #367 +/- ## =========================================== + Coverage 70.24% 70.66% +0.42% =========================================== Files 6 6 Lines 904 900 -4 =========================================== + Hits 635 636 +1 + Misses 269 264 -5 ``` | [Flag](https://app.codecov.io/gh/reportportal/agent-python-pytest/pull/367/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reportportal) | Coverage Δ | | |---|---|---| | [unittests](https://app.codecov.io/gh/reportportal/agent-python-pytest/pull/367/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reportportal) | `70.66% <80.00%> (+0.42%)` | :arrow_up: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=reportportal#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

HardNorth commented 2 months ago

@virdok Thanks!

virdok commented 2 months ago

Is there any expected date to release this fix in master in a new version? @HardNorth