search

pytest-dev / pytest

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing
https://pytest.org
MIT License
11.93k stars 2.65k forks source link

empty_parameter_set_mark = xfail + `--runxfail` => crash #4497

Open asottile opened 5 years ago

asottile commented 5 years ago

This started as a "hmm... that's weird" and ended at a crash. go figure.

This is super low priority and more of a result of "pytest has sooo many options!"

Today I learned that this generates a single test which is always skipped

import pytest

@pytest.mark.parametrize(
    ('a', 'b'),
    # no cases defined yet
    (

    ),
)
def test(a, b):
    pass
$ pytest t2.py 
============================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/t, inifile:
collected 1 item                                                               

t2.py s                                                                  [100%]

========================== 1 skipped in 0.01 seconds ===========================

But with a certain set of arguments:

$ pytest --runxfail -o empty_parameter_set_mark=xfail t2.py
============================= test session starts ==============================
platform linux -- Python 3.6.6, pytest-4.0.1, py-1.7.0, pluggy-0.8.0
rootdir: /tmp/t, inifile:
collected 1 item                                                               

t2.py E                                                                  [100%]

==================================== ERRORS ====================================
________________________ ERROR at setup of test[a0-b0] _________________________

self = <CallInfo when='setup' exception: 'SubRequest' object has no attribute 'param'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x7fe789d24840>
when = 'setup', treat_keyboard_interrupt_as_exception = False

    def __init__(self, func, when, treat_keyboard_interrupt_as_exception=False):
        #: context of invocation: one of "setup", "call",
        #: "teardown", "memocollect"
        self.when = when
        self.start = time()
        try:
>           self.result = func()

venv/lib/python3.6/site-packages/_pytest/runner.py:211: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
venv/lib/python3.6/site-packages/_pytest/runner.py:193: in <lambda>
    lambda: ihook(item=item, **kwds),
venv/lib/python3.6/site-packages/pluggy/hooks.py:284: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
venv/lib/python3.6/site-packages/pluggy/manager.py:67: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
venv/lib/python3.6/site-packages/pluggy/manager.py:61: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
venv/lib/python3.6/site-packages/_pytest/runner.py:114: in pytest_runtest_setup
    item.session._setupstate.prepare(item)
venv/lib/python3.6/site-packages/_pytest/runner.py:381: in prepare
    col.setup()
venv/lib/python3.6/site-packages/_pytest/python.py:1442: in setup
    fixtures.fillfixtures(self)
venv/lib/python3.6/site-packages/_pytest/fixtures.py:297: in fillfixtures
    request._fillfixtures()
venv/lib/python3.6/site-packages/_pytest/fixtures.py:470: in _fillfixtures
    item.funcargs[argname] = self.getfixturevalue(argname)
venv/lib/python3.6/site-packages/_pytest/fixtures.py:517: in getfixturevalue
    return self._get_active_fixturedef(argname).cached_result[0]
venv/lib/python3.6/site-packages/_pytest/fixtures.py:540: in _get_active_fixturedef
    self._compute_fixture_value(fixturedef)
venv/lib/python3.6/site-packages/_pytest/fixtures.py:626: in _compute_fixture_value
    fixturedef.execute(request=subrequest)
venv/lib/python3.6/site-packages/_pytest/fixtures.py:927: in execute
    return hook.pytest_fixture_setup(fixturedef=self, request=request)
venv/lib/python3.6/site-packages/pluggy/hooks.py:284: in __call__
    return self._hookexec(self, self.get_hookimpls(), kwargs)
venv/lib/python3.6/site-packages/pluggy/manager.py:67: in _hookexec
    return self._inner_hookexec(hook, methods, kwargs)
venv/lib/python3.6/site-packages/pluggy/manager.py:61: in <lambda>
    firstresult=hook.spec.opts.get("firstresult") if hook.spec else False,
venv/lib/python3.6/site-packages/_pytest/fixtures.py:969: in pytest_fixture_setup
    result = call_fixture_func(fixturefunc, request, kwargs)
venv/lib/python3.6/site-packages/_pytest/fixtures.py:828: in call_fixture_func
    res = fixturefunc(**kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

request = <SubRequest 'a' for <Function 'test[a0-b0]'>>

    def get_direct_param_fixture_func(request):
>       return request.param
E       AttributeError: 'SubRequest' object has no attribute 'param'

venv/lib/python3.6/site-packages/_pytest/fixtures.py:301: AttributeError
=========================== 1 error in 0.26 seconds ============================
gnikonorov commented 4 years ago

Can still reproduce. I'll try to have a PR up for this soon

gnikonorov commented 4 years ago

Issue comes down to --runxfail not taking precedent over empty_parameter_set_mark=xfail. If we make runxfail take precedent, the issue will go away.

Below is how pytest has interpreted the mark in the original issue report

pip install -e ../ && python3 -m pytest --runxfail  test_change.py
ValueError: Mark(name='skip', args=(), kwargs={'reason': "got empty parameter set ('a', 'b'), function test at /home/gnikonorov/pytest/gleb_test/test_change.py:2"})

pip install -e ../ && python3 -m pytest --runxfail -o empty_parameter_set_mark=xfail  test_change.py
ValueError: Mark(name='xfail', args=(), kwargs={'run': False, 'reason': "got empty parameter set ('a', 'b'), function test at /home/gnikonorov/pytest/gleb_test/test_change.py:2"})