Test collection order inconsistent when parameter options come from sets

[mthuurne]

Test collection fails for me when using pytest-xdist:

$ pytest -n10 
====================================================== test session starts ======================================================
platform linux -- Python 3.8.4, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /home/mth/bb/softfab, configfile: pyproject.toml
plugins: xdist-1.32.0, cov-2.10.1
gw0 [201] / gw1 [201] / gw2 [201] / gw3 [201] / gw4 [201] / gw5 [201] / gw6 [201] / gw7 [201] / gw8 [201] / gw9 [201]
collecting 0 items / 1 error                                                                                                    
============================================================ ERRORS =============================================================
_____________________________________________________ ERROR collecting gw0 ______________________________________________________
Different tests were collected between gw2 and gw0. The difference is:
--- gw2

+++ gw0

@@ -1,5 +1,5 @@

+tests/unit/test_commandline.py::test_create_user[user]
 tests/unit/test_commandline.py::test_create_user[guest]
-tests/unit/test_commandline.py::test_create_user[user]
 tests/unit/test_commandline.py::test_create_user[operator]
 tests/unit/test_commandline.py::test_create_user[None]
 tests/unit/test_commandline.py::test_remove_user

Similar diffs occur on the other workers as well.

There are more parameterized tests, but only test_create_user triggers this problem. The reason is that the four test cases after parameter substitution are not always in the same order because the parameter options originate from a set:

roleNames = {'guest', 'user', 'operator'}

@mark.parametrize('role', list(roleNames) + [None])
def test_create_user(cli, role):
    ...

If I change list(roleNames) to sorted(roleNames) or export PYTHONHASHSEED on the command line, the order of the options will always be the same and the problem goes away.

A possible solution would be to set PYTHONHASHSEED to the same value for all workers.

[RonnyPfannschmidt]

That's intentional, please use sorted

[mthuurne]

Are there any upsides to this behavior if it is intentional? Because it cost me some time to figure out what is going on here, so there are certainly downsides to it.

[RonnyPfannschmidt]

There are a few structural issues with xdist/pytest that imply that accidental random order tests will cause issues

One key issue is that test ids are not actually unique, so things are layered unfortunate

[mthuurne]

Since accidental random order can cause problems, wouldn't that be a motivation to force the same hash seed on all workers? It would remove an important source of randomness, after all.

[nicoddemus]

Since accidental random order can cause problems, wouldn't that be a motivation to force the same hash seed on all workers? It would remove an important source of randomness, after all.

That's a good idea, I don't think there are any downsides of doing that.

[RonnyPfannschmidt]

That's unclear, xdist supports invocations where hash order consistency is impossible to implement, I'd prefer pytest to guide towards permanent order not empheral fragile order

[zzzeek]

we are getting this failure suddenly after upgrading to pytest 6.1.0. With 6.0.2 and lower we do not get the error.

this is for the Alembic subproject of SQLAlchemy and we are sorting our tests.

The issue is not reproducible on every environment so I'm not sure I can reproduce it for you. im not sure of the cause yet.

[zzzeek]

something seems to be going wrong with the pathing with 6.1.0 of pytest.

We running from this directory:

/home/jenkins/workspace/alembic_gerrit/6a4aefb0

running one suite like this:

.tox/py37-pyoptimize-sqla13-oracle/bin/python3 -m pytest -n3   --write-idents db_idents.txt tests/test_command.py

with 6.0.2, the tests illustrate their path in relation to ., here's a partial listing:

gw1] [  1%] PASSED tests/test_command.py::CommandLineTest::test_config_file_default 
tests/test_command.py::CommandLineTest::test_help_text 
[gw2] [  2%] PASSED tests/test_command.py::CommandLineTest::test_config_file_env_variable 
[gw0] [  3%] PASSED tests/test_command.py::CommandLineTest::test_config_file_c_override 
tests/test_command.py::CommandLineTest::test_config_file_env_variable_c_override 
tests/test_command.py::CommandLineTest::test_init_file_doesnt_exist 
[gw2] [  4%] PASSED tests/test_command.py::CommandLineTest::test_init_file_doesnt_exist 
tests/test_command.py::CommandLineTest::test_init_w_package 
[gw1] [  5%] PASSED tests/test_command.py::CommandLineTest::test_help_text 
...

when we run with 6.1.0, it randomly places the previous path token:

[gw0] [  1%] PASSED 6a4aefb0/tests/test_command.py::CommandLineTest::test_config_file_c_override 
[gw1] [  2%] PASSED 6a4aefb0/tests/test_command.py::CommandLineTest::test_config_file_default 
6a4aefb0/tests/test_command.py::CommandLineTest::test_help_text 
[gw2] [  3%] PASSED 6a4aefb0/tests/test_command.py::CommandLineTest::test_config_file_env_variable 
6a4aefb0/tests/test_command.py::CommandLineTest::test_init_file_doesnt_exist 
..

when we get the conflict, it's because one worker includes the 6a4aefb0 prefix and another one does not.

I have no idea as of yet what mechanism is causing it to get those paths. still investigating.

[zzzeek]

the bug is in pytest 6.1.0, unless this is intentional, but this seems to be related to the issue.

the ".nodeid" of an item now has an extra path token in it:

here's inside of def pytest_collection_modifyitems(session, config, items):

Here's 6.1.0:

(Pdb) items[0]
<Function test_history_full>
(Pdb) items[0].nodeid
'6a4aefb0/tests/test_command.py::HistoryTest::test_history_full'

here's 6.0.2:

(Pdb) items[0].nodeid
'tests/test_command.py::HistoryTest::test_history_full'

why did that path change? Why is it storing a superdirectory like that?

[zzzeek]

this is the commit that breaks it:

https://github.com/pytest-dev/pytest/commit/70f3ad1c1f31b35d4004f92734b4afd6c8fbdecf

on my CI environment I'm using the Jenkins matrix build plugin. When the tests run, one path upwards is a checkout of the source, that includes setup.cfg, pyproject.toml, etc. the change above is causing it to think I'm inside of a path inside the checkout. still looking...

[zzzeek]

OK, figured it out, unrelated to this issue and I can of course work around it by supplying --rootdir, will report the bug at pytest, though it also seems that pytest-xdist behaves inconsistently for each worker with the new locate_config() logic as well.

[nicoddemus]

Hi @zzzeek,

Indeed there was a bug in pytest related to that commit, can you try 6.1.1 and confirm that it fixed this issue as well?

[zzzeek]

sure, I can't make the above failure happen with 6.1.0 anymore as the actual pytest-xdist part of it seemed to rely upon some FS specifics but the pathing is correct again with 6.1.1.