Does not work with parametrized nodeids

[blueyed]

pytest tests/test_database.py::TestDatabaseFixtures::test_access\[db\] works, but adding --count 10 fails:

collecting 0 items
ERROR: not found: …/Vcs/pytest-django/tests/test_database.py::TestDatabaseFixtures::test_access[db]
(no name '…/Vcs/pytest-django/tests/test_database.py::TestDatabaseFixtures::test_access[db]' in any of [<Instance '()'>])

pytest version 3.6.3.dev5+gb7b9c54d

[RonnyPfannschmidt]

does this happen in older pytest versions ?

[vbarbaresi]

It happens in older pytest versions as well (at least until 2.8) I opened an issue on pytest tracker https://github.com/pytest-dev/pytest/issues/4142 providing a unit test reproducing the bug.

[blueyed]

I wonder if this is also related to adding/removing ::() (https://github.com/pytest-dev/pytest/issues/4127#issuecomment-429363005).

[blueyed]

According to https://github.com/pytest-dev/pytest/issues/4142 it likely is not.

[1Mark]

Any progress on this?

[vyahello]

Hey, looks like this happens on the latest pytest version Do we have any updates on this one?

[DavidAntliff]

This is happening for me with pytest 7.1.0 and pytest-repeat 0.9.1.

Is it related to the way test items are renamed and collected, in that pytest-repeat modifies the name with extra parameters, and this affects name matching?

For example, with a parametrized test that has one parameter of value 256, this works (runs once):

 $ pytest test.py::test_something[256]

And this works (runs 1000 times but for each parameter, which may be far too many tests):

 $ pytest test.py::test_something --count 1000

But to run for a single parameter value, this does not work:

 $ pytest test.py::test_something[256] --count 1000
(no name 'test.py::test_something[256]' in any of [<Module test.py>])

And this runs once, but does not repeat:

 $ pytest test.py::test_something[256-1-1000] --count 1000

Is there a known workaround, aside from running pytest over and over in a shell script loop?

[DavidAntliff]

I'm curious if there's been any thinking from the contributors as to how this might work, if we could get it to work?

I come across this so often I'd be happy to help fix it, if I could get in sync with the designers' thoughts...

[RonnyPfannschmidt]

Currently no design work on this was done

I believe major internals Changes in pytest itself are necessary to enable this

[okken]

Seems to work fine now, with pytest 7.4.2 and pytest-repeat 0.9.2

import pytest

@pytest.mark.parametrize('a', (10, 20))
def test_foo(a):
    ...

test run:

(pytest-repeat) $ pytest test_param.py -v --count 3
================================= test session starts =================================
collected 6 items                                                                     

test_param.py::test_foo[10-1-3] PASSED                                          [ 16%]
test_param.py::test_foo[10-2-3] PASSED                                          [ 33%]
test_param.py::test_foo[10-3-3] PASSED                                          [ 50%]
test_param.py::test_foo[20-1-3] PASSED                                          [ 66%]
test_param.py::test_foo[20-2-3] PASSED                                          [ 83%]
test_param.py::test_foo[20-3-3] PASSED                                          [100%]

================================== 6 passed in 0.01s ==================================

[DavidAntliff]

@okken I think the issue remains and it’s that you can’t do this:

 $ pytest test_param.py::test_foo[10] -v --count 3

[RonnyPfannschmidt]

This one is unfixable without a major change in pytest

[okken]

The act of using --count 3 changes the nodeids, so with --count 3, test_foo[10] no longer exists.

There is a workaround, selecting parameters by keyword instead of nodeid:

(pytest-repeat) $ pytest test_param.py -v -k 'test_foo[10' --count 3
=========================== test session starts ============================
collected 6 items / 3 deselected / 3 selected                              

test_param.py::test_foo[10-1-3] PASSED                               [ 33%]
test_param.py::test_foo[10-2-3] PASSED                               [ 66%]
test_param.py::test_foo[10-3-3] PASSED                               [100%]

===================== 3 passed, 3 deselected in 0.01s ======================

[okken]

We also cannot select individual counts using nodeid:

(pytest-repeat) $ pytest 'test_param.py::test_foo[10-2-3]'              
=========================== test session starts ============================
collected 0 items                                                          

========================== no tests ran in 0.00s ===========================
ERROR: not found: /Users/okken/projects/pytest-repeat/test_param.py::test_foo[10-2-3]
(no name '/Users/okken/projects/pytest-repeat/test_param.py::test_foo[10-2-3]' in any of [<Module test_param.py>])

I don't think this is a bug, it's an affect of when the counts are added to the nodeid.

And there is the same workaround, use keywords.

(pytest-repeat) $ pytest test_param.py -v -k 'test_foo[10-2-3' --count 3
=========================== test session starts ============================
collected 6 items / 5 deselected / 1 selected                              

test_param.py::test_foo[10-2-3] PASSED                               [100%]

===================== 1 passed, 5 deselected in 0.00s ======================

[okken]

@RonnyPfannschmidt this brings me to a question of "why leave this issue open?"

• We know desired syntax isn't realistic due to the implementation of both pytest-repeat and more importantly, pytest.
• So unless pytest changes, it's not fixable.
• There is a workaround, use keywords.

So why keep this open?

[okken]

Also just swapped "bug" for "enhancement", because there is a way to do the requested action, just not with the desired syntax.

[RonnyPfannschmidt]

I'd like to turn this into a feature request for pytest

[okken]

I personally wouldn't know how to describe the feature request. Partly because I don't fully understand the limitations of pytest that are the problem. But I agree.

I do think, at the very least, this issue should remain open until the workaround is documented and easy to find.

[okken]

@DavidAntliff Does the -k workaround work for your use case?

[DavidAntliff]

@DavidAntliff Does the -k workaround work for your use case?

Yes, that works for me - thank you. I've been looking for a way to do this for a while.