And indeed I expect pytest to fail if I have some randomness in my test definitions (say a random parameterized fixture).
Different tests were collected between gw1 and gw2. The difference is:
...
But I noticed that pytest exits with code 1 in this case. This code is reserved for "collected but failed" https://docs.pytest.org/en/stable/reference/exit-codes.html. I think this should be either code 3 or 5, but at least not code 1 since this feels like a collection or otherwise "internal" error.
I noticed https://pytest-xdist.readthedocs.io/en/stable/known-limitations.html#order-and-amount-of-test-must-be-consistent
And indeed I expect pytest to fail if I have some randomness in my test definitions (say a random parameterized fixture).
But I noticed that pytest exits with code 1 in this case. This code is reserved for "collected but failed" https://docs.pytest.org/en/stable/reference/exit-codes.html. I think this should be either code 3 or 5, but at least not code 1 since this feels like a collection or otherwise "internal" error.