search

allure-framework / allure-python

Allure integrations for Python test frameworks
https://allurereport.org/
Apache License 2.0
730 stars 238 forks source link

Regression in allure-pytest 2.13.0: crash in allure_title when called over pytest.Item with no .obj #733

Closed mshonichev closed 1 year ago

mshonichev commented 1 year ago

I'm submitting a ...

What is the current behavior?

When running pytest example of collecting yaml tests, allure-pytest 2.13.0 crashes on allure_title

    def allure_title(item):
>       return getattr(item.obj, "__allure_display_name__", None)
E       AttributeError: 'YamlItem' object has no attribute 'obj'

It seems legitimate for pytest.Item to not have .obj property during lifecycle.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

python -m venv .venv
.venv/bin/activate
(.venv) pip install pytest==7.2.1 allure-pytest==2.13.0

Follow the example of collecting Yaml tests, create conftest.py, test_simple.yaml https://docs.pytest.org/en/stable/example/nonpython.html#non-python-tests

run yaml tests with allure

% python -m py.test test_simple.yaml --alluredir=allure-results
================================================================================================= test session starts ==================================================================================================
platform darwin -- Python 3.9.2, pytest-7.2.1, pluggy-0.13.1
rootdir: /Users/m.shonichev/src/yadro/build-tools/tests/yaml
plugins: allure-pytest-2.13.0, timeout-2.0.1
collected 2 items                                                                                                                                                                                                      

test_simple.yaml EE                                                                                                                                                                                              [100%]

======================================================================================================== ERRORS ========================================================================================================
___________________________________________________________________________________________ ERROR at setup of usecase: hello ___________________________________________________________________________________________

cls = <class '_pytest.runner.CallInfo'>, func = <function call_runtest_hook.<locals>.<lambda> at 0x104e0cd30>, when = 'setup', reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.

        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

../../../../venv/lib/python3.9/site-packages/_pytest/runner.py:339: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>       lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    )

../../../../venv/lib/python3.9/site-packages/_pytest/runner.py:260: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_HookCaller 'pytest_runtest_setup'>, args = (), kwargs = {'item': <YamlItem hello>}, notincall = set()

    def __call__(self, *args, **kwargs):
        if args:
            raise TypeError("hook calling supports only keyword arguments")
        assert not self.is_historic()
        if self.spec and self.spec.argnames:
            notincall = (
                set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
            )
            if notincall:
                warnings.warn(
                    "Argument(s) {} which are declared in the hookspec "
                    "can not be found in this hook call".format(tuple(notincall)),
                    stacklevel=2,
                )
>       return self._hookexec(self, self.get_hookimpls(), kwargs)

../../../../venv/lib/python3.9/site-packages/pluggy/hooks.py:286: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.config.PytestPluginManager object at 0x103db0220>, hook = <_HookCaller 'pytest_runtest_setup'>
methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/Users/m.shonichev/src/venv/lib/python3.9/site-pack...>, <HookImpl plugin_name='allure_listener', plugin=<allure_pytest.listener.AllureListener object at 0x104e4b310>>, ...]
kwargs = {'item': <YamlItem hello>}

    def _hookexec(self, hook, methods, kwargs):
        # called from all hookcaller instances.
        # enable_tracing will set its own wrapping function at self._inner_hookexec
>       return self._inner_hookexec(hook, methods, kwargs)

../../../../venv/lib/python3.9/site-packages/pluggy/manager.py:93: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

hook = <_HookCaller 'pytest_runtest_setup'>
methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/Users/m.shonichev/src/venv/lib/python3.9/site-pack...>, <HookImpl plugin_name='allure_listener', plugin=<allure_pytest.listener.AllureListener object at 0x104e4b310>>, ...]
kwargs = {'item': <YamlItem hello>}

>   self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
        methods,
        kwargs,
        firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    )

../../../../venv/lib/python3.9/site-packages/pluggy/manager.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/Users/m.shonichev/src/venv/lib/python3.9/site-pack...>, <HookImpl plugin_name='allure_listener', plugin=<allure_pytest.listener.AllureListener object at 0x104e4b310>>, ...]
caller_kwargs = {'item': <YamlItem hello>}, firstresult = False

    def _multicall(hook_impls, caller_kwargs, firstresult=False):
        """Execute a call into multiple python functions/methods and return the
        result(s).

        ``caller_kwargs`` comes from _HookCaller.__call__().
        """
        __tracebackhide__ = True
        results = []
        excinfo = None
        try:  # run impl and wrapper setup functions in a loop
            teardowns = []
            try:
                for hook_impl in reversed(hook_impls):
                    try:
                        args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                    except KeyError:
                        for argname in hook_impl.argnames:
                            if argname not in caller_kwargs:
                                raise HookCallError(
                                    "hook call must provide argument %r" % (argname,)
                                )

                    if hook_impl.hookwrapper:
                        try:
                            gen = hook_impl.function(*args)
                            next(gen)  # first yield
                            teardowns.append(gen)
                        except StopIteration:
                            _raise_wrapfail(gen, "did not yield")
                    else:
                        res = hook_impl.function(*args)
                        if res is not None:
                            results.append(res)
                            if firstresult:  # halt further impl calls
                                break
            except BaseException:
                excinfo = sys.exc_info()
        finally:
            if firstresult:  # first result hooks return a single value
                outcome = _Result(results[0] if results else None, excinfo)
            else:
                outcome = _Result(results, excinfo)

            # run all wrapper post-yield blocks
            for gen in reversed(teardowns):
                try:
>                   gen.send(outcome)

../../../../venv/lib/python3.9/site-packages/pluggy/callers.py:203: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <allure_pytest.listener.AllureListener object at 0x104e4b310>, item = <YamlItem hello>

    @pytest.hookimpl(hookwrapper=True)
    def pytest_runtest_setup(self, item):
        if not self._cache.get(item.nodeid):
            uuid = self._cache.push(item.nodeid)
            test_result = TestResult(name=item.name, uuid=uuid, start=now(), stop=now())
            self.allure_logger.schedule_test(uuid, test_result)
        yield
        self._update_fixtures_children(item)
        uuid = self._cache.get(item.nodeid)
        test_result = self.allure_logger.get_test(uuid)
        params = item.callspec.params if hasattr(item, 'callspec') else {}
>       test_result.name = allure_name(item, params)

../../../../venv/lib/python3.9/site-packages/allure_pytest/listener.py:99: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

item = <YamlItem hello>, parameters = {}

    def allure_name(item, parameters):
        name = item.name
>       title = allure_title(item)

../../../../venv/lib/python3.9/site-packages/allure_pytest/utils.py:111: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

item = <YamlItem hello>

    def allure_title(item):
>       return getattr(item.obj, "__allure_display_name__", None)
E       AttributeError: 'YamlItem' object has no attribute 'obj'

../../../../venv/lib/python3.9/site-packages/allure_pytest/utils.py:30: AttributeError
____________________________________________________________________________________________ ERROR at setup of usecase: ok _____________________________________________________________________________________________

cls = <class '_pytest.runner.CallInfo'>, func = <function call_runtest_hook.<locals>.<lambda> at 0x10502d820>, when = 'setup', reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.

        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

../../../../venv/lib/python3.9/site-packages/_pytest/runner.py:339: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>       lambda: ihook(item=item, **kwds), when=when, reraise=reraise
    )

../../../../venv/lib/python3.9/site-packages/_pytest/runner.py:260: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_HookCaller 'pytest_runtest_setup'>, args = (), kwargs = {'item': <YamlItem ok>}, notincall = set()

    def __call__(self, *args, **kwargs):
        if args:
            raise TypeError("hook calling supports only keyword arguments")
        assert not self.is_historic()
        if self.spec and self.spec.argnames:
            notincall = (
                set(self.spec.argnames) - set(["__multicall__"]) - set(kwargs.keys())
            )
            if notincall:
                warnings.warn(
                    "Argument(s) {} which are declared in the hookspec "
                    "can not be found in this hook call".format(tuple(notincall)),
                    stacklevel=2,
                )
>       return self._hookexec(self, self.get_hookimpls(), kwargs)

../../../../venv/lib/python3.9/site-packages/pluggy/hooks.py:286: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <_pytest.config.PytestPluginManager object at 0x103db0220>, hook = <_HookCaller 'pytest_runtest_setup'>
methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/Users/m.shonichev/src/venv/lib/python3.9/site-pack...>, <HookImpl plugin_name='allure_listener', plugin=<allure_pytest.listener.AllureListener object at 0x104e4b310>>, ...]
kwargs = {'item': <YamlItem ok>}

    def _hookexec(self, hook, methods, kwargs):
        # called from all hookcaller instances.
        # enable_tracing will set its own wrapping function at self._inner_hookexec
>       return self._inner_hookexec(hook, methods, kwargs)

../../../../venv/lib/python3.9/site-packages/pluggy/manager.py:93: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

hook = <_HookCaller 'pytest_runtest_setup'>
methods = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/Users/m.shonichev/src/venv/lib/python3.9/site-pack...>, <HookImpl plugin_name='allure_listener', plugin=<allure_pytest.listener.AllureListener object at 0x104e4b310>>, ...]
kwargs = {'item': <YamlItem ok>}

>   self._inner_hookexec = lambda hook, methods, kwargs: hook.multicall(
        methods,
        kwargs,
        firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
    )

../../../../venv/lib/python3.9/site-packages/pluggy/manager.py:84: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

hook_impls = [<HookImpl plugin_name='nose', plugin=<module '_pytest.nose' from '/Users/m.shonichev/src/venv/lib/python3.9/site-pack...>, <HookImpl plugin_name='allure_listener', plugin=<allure_pytest.listener.AllureListener object at 0x104e4b310>>, ...]
caller_kwargs = {'item': <YamlItem ok>}, firstresult = False

    def _multicall(hook_impls, caller_kwargs, firstresult=False):
        """Execute a call into multiple python functions/methods and return the
        result(s).

        ``caller_kwargs`` comes from _HookCaller.__call__().
        """
        __tracebackhide__ = True
        results = []
        excinfo = None
        try:  # run impl and wrapper setup functions in a loop
            teardowns = []
            try:
                for hook_impl in reversed(hook_impls):
                    try:
                        args = [caller_kwargs[argname] for argname in hook_impl.argnames]
                    except KeyError:
                        for argname in hook_impl.argnames:
                            if argname not in caller_kwargs:
                                raise HookCallError(
                                    "hook call must provide argument %r" % (argname,)
                                )

                    if hook_impl.hookwrapper:
                        try:
                            gen = hook_impl.function(*args)
                            next(gen)  # first yield
                            teardowns.append(gen)
                        except StopIteration:
                            _raise_wrapfail(gen, "did not yield")
                    else:
                        res = hook_impl.function(*args)
                        if res is not None:
                            results.append(res)
                            if firstresult:  # halt further impl calls
                                break
            except BaseException:
                excinfo = sys.exc_info()
        finally:
            if firstresult:  # first result hooks return a single value
                outcome = _Result(results[0] if results else None, excinfo)
            else:
                outcome = _Result(results, excinfo)

            # run all wrapper post-yield blocks
            for gen in reversed(teardowns):
                try:
>                   gen.send(outcome)

../../../../venv/lib/python3.9/site-packages/pluggy/callers.py:203: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <allure_pytest.listener.AllureListener object at 0x104e4b310>, item = <YamlItem ok>

    @pytest.hookimpl(hookwrapper=True)
    def pytest_runtest_setup(self, item):
        if not self._cache.get(item.nodeid):
            uuid = self._cache.push(item.nodeid)
            test_result = TestResult(name=item.name, uuid=uuid, start=now(), stop=now())
            self.allure_logger.schedule_test(uuid, test_result)
        yield
        self._update_fixtures_children(item)
        uuid = self._cache.get(item.nodeid)
        test_result = self.allure_logger.get_test(uuid)
        params = item.callspec.params if hasattr(item, 'callspec') else {}
>       test_result.name = allure_name(item, params)

../../../../venv/lib/python3.9/site-packages/allure_pytest/listener.py:99: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

item = <YamlItem ok>, parameters = {}

    def allure_name(item, parameters):
        name = item.name
>       title = allure_title(item)

../../../../venv/lib/python3.9/site-packages/allure_pytest/utils.py:111: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

item = <YamlItem ok>

    def allure_title(item):
>       return getattr(item.obj, "__allure_display_name__", None)
E       AttributeError: 'YamlItem' object has no attribute 'obj'

../../../../venv/lib/python3.9/site-packages/allure_pytest/utils.py:30: AttributeError
=============================================================================================== short test summary info ================================================================================================
ERROR test_simple.yaml::hello - AttributeError: 'YamlItem' object has no attribute 'obj'
ERROR test_simple.yaml::ok - AttributeError: 'YamlItem' object has no attribute 'obj'
================================================================================================== 2 errors in 0.23s ===================================================================================================

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

Other information

The bug is not reproducing with allure-pytest 2.12.0

The commit that broke it:

https://github.com/allure-framework/allure-python/commit/86cff25b351c7ad63af3136f0059228b78dfe461#diff-fefb374ddcddaa062601d8d7f5b15f0a5d80552dea44452f34eec87a3a9bada7

Note, that such issue was already fixed in https://github.com/vgorkavenko/allure-python/commit/bdce11f2de1059c22cdb459b1d05526bf240cac5

As a reaction to issue with mypy: https://github.com/allure-framework/allure-python/issues/542

Reverting mentioned commit in 2.13 is clearly the reason regression was introduced.

I believe further testing would reveal that bug with mypy again reintroduced too.

delatrie commented 1 year ago

Hi, @mshonichev !

Thank you for the detailed report! Your observations are correct, that was wrong of me not to check if the obj attr is guaranteed to exists.

I've already created the PR. The fix will be released soon.