search

klrmn / pytest-random

randomize your py.test run
20 stars 10 forks source link

Random tests should run in the same relative order when some are added or removed. #11

Open kevincox opened 8 years ago

kevincox commented 8 years ago

Currently if you have a list of tests that you are running and you add or remove one the entire list gets shuffled differently. It would be much nicer if adding or removing tests kept the same random order. Effectively adding or removing tests from the list.

This makes debugging very difficult as you can't easily narrow down the lists of tests to find the one that is causing the problem.

kevincox commented 8 years ago

FWIW here is an implementation. However it should use an actual consistent hashing function rather then python's builtin hash which I don't think promises to be consistent. However it was enough for my problem. Just replace the old shuffle call with this line.

seed = str(config.option.random_seed)                                                           
def key(item):                                                                                  
    return hash(seed + str(hash(item)))

items.sort(key=key)