Currently bucketing is done on the hash of the name. I think it would be more fair if something along these lines happened:
(This is of course pseudocode but you get the idea)
buckets = collections.defaultdict(list)
# Sort by number of tests, then test name or hash of test name (like the current implementation does) (so it is deterministic)
test_cases = sorted(
test_cases,
key=lambda test_case: (test_case.number_of_tests, test_case.__name__),
)
# Round robin insert into buckets
test_cases_iter = iter(test_cases)
try:
while True:
for bucket_number in xrange(bucket_count):
buckets[bucket_number].append(test_cases_iter.next())
except StopIteration:
# Cool we're done
pass
Currently bucketing is done on the hash of the name. I think it would be more fair if something along these lines happened:
(This is of course pseudocode but you get the idea)