Open mbkroese opened 3 years ago
I assume it's not deterministic here because of set
, or can you repro it also with list
or tuple
?
No, this problem occurs when either the data structure has non-deterministic order or the code generating the parametrised test cases is for some reason not deterministic.
I think we could go with 1.
aka make sure the tests inside same parametrize
are run in the same group. However, the downside is ofc that if one parametrised test is very time consuming vs the rest of the suite, the splits would not be great.
OTOH, maybe it's better to make sure that we don't accidentally skip tests (or run some test in multiple groups) 🤔
With these thoughts, I'd go with 1.
🙂
downside is ofc that if one parametrised test is very time consuming vs the rest of the suite, the splits would not be great.
Yes, and I wonder if we should perhaps be safe by default (i.e. option 1) and allow users to do the unsafe thing (existing behaviour). If we make clear what the tradeoffs are, the user can then decide for him/herself. In other words add a parameter that --split-level=func
by default but can be set to --split-level=parametrized
or --split-level=file
?
Was this implemented yet? If not I think it would be a great feature to have, as I have some tests I want ran in the same group under the same parametrize decorator
The big assumption underlying the two splitting algorithms is that the order of collected items is constant. However, I've come across a case where this assumption was violated. In my case I had a test parametrised with
pytest.mark.parametrize
, but the items to parametrize with would sometimes change order.Take this example:
If you run this often enough you'll see that the order changes:
and
I'm not sure how to address this, but I think there are a few options:
parametrize
for the same test. In other words, make sure that a single group will run all tests fortest_hello
.pytest
with those pre-calculated groups (so not really using this plugin as a plugin :p)