Open Alex-Chizhov opened 5 years ago
As I understand it, xdist started test 2 times, and the test data generator also worked 2 times. Xdist doesn't like it. And we have to use a static JSON file with test data. We need to make the generation of this data in the fixture, not in the body of the test.
I am also running into this issue when using tests that do fuzzing. It would be very useful if there was a way to disable the "generating tests twice must produce identical tests" check. For example, given this test code in a file example.py
:
@pytest.mark.parametrize('x,y', [
(random.random(), random.random())
])
def test_example(x, y):
assert isinstance(x, float)
assert isinstance(y, float)
This command passes:
pytest example.py
And this command fails:
pytest example.py -n 2
With an error similar to this one:
Different tests were collected between gw0 and gw1. The difference is:
--- gw0
+++ gw1
@@ -1 +1 @@
-example.py::test_example[0.16380351559829032-0.38206603085139057]
+example.py::test_example[0.2613173472636646-0.7205939052389861]
Oddly, with one parameter instead of two the test will pass even with -n 2
.
Hi,
Sorry for the delay!
As I understand it, xdist started test 2 times, and the test data generator also worked 2 times. Xdist doesn't like it. And we have to use a static JSON file with test data. We need to make the generation of this data in the fixture, not in the body of the test.
Yes, each worker performs a standard collection, and sends the collected test ids (in order) back to the master node. The master node ensures every worker collected the same number of tests and in the same order, because the scheduler will from that point on send just the test indexes (not the entire node id) to each worker to tell them which test to execute. That's why the collection must be the same across all workers.
"generating tests twice must produce identical tests" check
As explained above, unfortunately it is not a simple check, but a central design point how the scheduler and workers interact.
Hi, what is the solution to this? I also have:
@pytest.mark.parametrize("env", Envs.get_all_latest_envs())
def test_envs_all(env):
gym.make(env)
where I am trying to test an agent on a bunch of environments. Different tests were collected between gw0 and gw1. Envs.get_all_latest_envs()
just returns a list of env names. This works in the normal pytest execution, however some environments take forever to init, so I'd like to run others in parallel instead of waiting.
I'm getting a similar error.
plugins: xdist-1.29.0, forked-1.0.2, asyncio-0.10.0
gw0 I / gw1 I
gw0 [483] / gw1 [483]
gw1:None (gw1)
The difference is:
--- gw0
+++ gw1
@@ -1,483 +1,483 @@
+fast_rl/tests/test_Envs.py::test_envs_all[AmidarNoFrameskip-v40]
+fast_rl/tests/test_Envs.py::test_envs_all[AirRaidNoFrameskip-v40]
+fast_rl/tests/test_Envs.py::test_envs_all[maze-sample-5x5-v0]
+fast_rl/tests/test_Envs.py::test_envs_all[VideoPinballNoFrameskip-v40]
+fast_rl/tests/test_Envs.py::test_envs_all[AsteroidsNoFrameskip-v40]
@josiahls Try using sorted(Env.get_all_latest_envs())
. I had the same problem and the issue was each worker generating the "same" list but in a different order.
edit, moved my comment to a new issue #472
Had this issue, fixed with sorting the parameters
@pytest.mark.parametrize("foo", sorted(foos))
I have also been having this issue with parametrized tests, I found discussions here, which sounded promising, but for some reason sorting the params does not fix it.
Possibly relevant :
Relevant to this PR at present .. https://github.com/SciTools/iris/pull/4960 A sample failing actions job
Main package versions Using :
pytest 7.1.3 py38h578d9bd_0 conda-forge
pytest-forked 1.4.0 pyhd8ed1ab_0 conda-forge
pytest-xdist 2.5.0 pyhd8ed1ab_0 conda-forge
Basically I'm stumped -- I'm not clear why sorting didn't seem to solve it, possibly something else is going on. We could really use a hand with this, it's a serious sticking point -- we need xdist for the reduced CI testing time.
for some reason sorting the params does not fix it.
UPDATE: in addition to sorted parameters, tried numbering the tests (which are in a common class), so that their alphabetic order is now the same as the order of definitions in the class. And that just worked :open_mouth:
also stumbled over this issue. This problem is still there. Is there a workaround?
also stumbled over this issue. This problem is still there. Is there a workaround?
Ah, I have just installed pytest-randomly, and it works.
also stumbled over this issue. This problem is still there. Is there a workaround?
Ah, I have just installed pytest-randomly, and it works.
Stumbled on this issue and with pytest-randomly installed, it works fine.
@nicoddemus i believe the only action we can take is to document that if people use random objects as test id's its their responsibility to ensure consistent test ids, and pytest-randomly
@nicoddemus i believe the only action we can take is to document that if people use random objects as test id's its their responsibility to ensure consistent test ids, and pytest-randomly
I have a valid case where it can happen, can't we disable this check ?
There's no valid case to have randomized test ids
I have a simple test with parametrization:
And simple generator of test data for parametrezation:
When i launch test with additional argument:
-n 2
I get this error message:Tell me please what could be the problem