search

Akuli / porcupine

A decent editor written in tkinter
MIT License
146 stars 47 forks source link

Add the `pytest-randomly` Pytest Plugin #1403

Open benjamin-kirkbride opened 11 months ago

benjamin-kirkbride commented 11 months ago

As I thought through https://github.com/Akuli/porcupine/issues/1401 it occurred to me that there might be some incidental dependence on the order of tests. I looked into what solutions existed for this and pytest-randomly seems to be the most popular one. It also supports some nice features for selecting a specific random seed which may help with other flaky tests in time.

To implement, literally all that needs done is to install it and it uses it by default.

To disable once installed: pytest -p no:randomly.

I have ran the test suite several times locally using it and everything no tests fail, which is great. I notice no difference in test collection time.

Akuli commented 11 months ago

I have mixed feelings about this. I like the idea, but I also hate it when the CI fails randomly. For example, if a specific order of 3 tests causes a problem, that problem would only occur in about 17% of all test runs.

Akuli commented 11 months ago

Sorry, I somehow missed that you mentioned specifying a seed. Looking at the project's README it seems to print the seed when it starts, so it would be easy to reproduce order-dependent failures.

I think this seems like a good idea and we should try it :)

benjamin-kirkbride commented 11 months ago

Yep!

image

The nice thing is that seed is also shared with random.seed(), so will (in theory) actually help with the "CI fails randomly" issue. (I don't think we use random anywhere today, but in the future something may, including a dependency we are unaware of)