search

pytest-dev / pytest-random-order

pytest plugin to randomise the order of tests with some control over the randomness
MIT License
65 stars 10 forks source link

Suggestion: Add a warning to the README that default random seed is now fixed #38

Open liiight opened 5 years ago

liiight commented 5 years ago

After I installed this plugin I began seeing bizarre issues in my tests, like API returning 409 conflict claiming duplicate object creations. It took me a second and some time digging into the logs to figure out that my previous randomness that was in-charge of generating random object identifiers, is no longer as random as I though.

I then connected the dots and figured out that it was the doing of this plugin. I tried changing the scope of the bucket to the minimal level, class, but while the number of these issues dramatically decreased, they still occurred, as I have tests within the same class that require separate random seeds.

I am by no means "blaming" you for this as this plugin worked as it is designed and it is I who didn't fully read and/or understood the implications of using it. I am grateful for its creation and I'm sure many people get high value from it. I do have some suggestions in order to avoid lazy people like me in the future:

  1. Add a noticeable warning that this plugin manipulates the random seed and that it may disturb tests that rely on the random module.
  2. If I am not mistaken, the default random bucket scope is global. Perhaps it would be wise to either set it to the minimum scope by default or even not have a default at all, forcing the user to choose a scope so he'll have some understanding at least on the implications.

Again, I am very grateful for this plugin and I hope my suggestion are taken in the spirit they are written.

jbasko commented 5 years ago

Hi, thanks for the message.

By default, there's no randomisation anymore unless you enable it in one of three ways via CLI: --random-order (which will set module bucket type), --random-order-seed [seed] (which will set module bucket type unless you also specify --random-order-bucket=[bucket_type]), or --random-order-bucket=[bucket_type]. If you're referring to "by-default-once-it's-enabled" then it's module.

Re:warning, thanks for the suggestion, will add it the next time when I touch this.

I think the approach we have here is incompatible with one using custom seeds in their own tests. Just thinking loud, maybe we should do random.getstate() before every test case and then set it back after the test case with random.setstate().

xmo-odoo commented 5 years ago

@jbasko couldn't the plugin just use its own random.Random() instance, independent from the global one? the functions from the random module are pretty much just proxies to the corresponding method of a global Random instance (random._inst)