search

mtreinish / stestr

A parallel Python test runner built around subunit
Apache License 2.0
27 stars 33 forks source link

Test are run out of order when using `--load-list` #330

Closed Akrog closed 2 years ago

Akrog commented 2 years ago

Issue description

When using --load-list to run tests, the tests run in a different order than the one provided in the file, even when the --random argument is not present.

This worked with os-testr and I think it even worked with stestr at some point before sets were used to remove duplicates.

Expected behavior and actual behavior

The expected behavior is that the order of tests in the file is preserved unless the --random argument is provided.

Steps to reproduce the problem

We create file testfile with contents:

stestr.tests.test_return_codes.TestReturnCodes.test_history_show_passing
stestr.tests.test_return_codes.TestReturnCodesToxIni.test_serial_fails_from_func
stestr.tests.test_load.TestLoadCommand.test_empty_with_pretty_out

Now we run the 2 tests with:

$ tox -epy39 -- --serial --load-list testfile

And it will most likely run in a different order.