feature request: order tests by slowest-first

[charles-cooper]

in long running test suites (especially with xdist), it's useful to run the slowest tests first so that you end up with better load. the following code seems to do so:

def pytest_collection_modifyitems(config, items):
    try:
        with open(".test_durations") as f:
            s = f.read()
            durations = json.loads(s)
    except FileNotFoundError:
        durations = {}

    timings = {}
    for item in items:
        path, _, reqname = item.location
        request_id = f"{path}::{reqname}"
        timing = durations.get(request_id, None)
        if timing is not None:
            timings[item] = timing

    meantime = sum(timings.values()) / len(timings)
    # sort by highest time first
    items.sort(key=lambda item: -timings.get(item, meantime))

but i was thinking it might be good for this to be available in the pytest-split plugin itself, so we don't need to add this same function across different projects.

alternatively, i was thinking publishing this as a plugin myself, if pytest-split does not want to add it.

[jerry-git]

Hey 👋 ! There's already support for different splitting algorithms, see

• --splitting-algorithm CLI flag
• currently supported algos in the README
• and their implementations in https://github.com/jerry-git/pytest-split/blob/master/src/pytest_split/algorithms.py

I believe what you are asking is actually quite close to "least duration" algo that already exists.

Happy to take a contribution which introduces "slowest first" splitting algorithm 🙂

[charles-cooper]

yea these are interesting. the use case i am imagining is actually most useful with --splits 1 --group 1 because it reorders the items on the local worker. do you think "slowest first" makes sense with more splits?

[jerry-git]

If the main motivation is to optimise the usage with xdist, I guess there could be also use cases which prefer to have multiple test runner nodes (e.g. GHA workers). So, pytest-split for splitting the work to multiple nodes and then xdist for splitting the work inside each node.