pytest-replay does not respect the order of the replay file

[DavideCanton]

Currently, pytest-replay does not respect the order in the replay file, but it just filters the tests in the order collected by pytest:

λ cat replay.txt
{"nodeid": "tests/foo_test.py::test_foo_1"}
{"nodeid": "tests/bar_test.py::test_bar_1"}

λ pytest --replay=replay.txt -v
================ test session start ==============
platform win32 -- Python 3.10.12, pytest-7.4.4, pluggy-1.3.0 -- C:\Users\CantonDavide\source\prove\prova_pytest\.venv\Scripts\python.exe
cachedir: .pytest_cache
rootdir: C:\Users\CantonDavide\source\prove\prova_pytest
plugins: replay-1.4.0
collected 4 items / 2 deselected / 2 selected

tests/bar_test.py::test_bar_1 PASSED                 [ 50%]
tests/foo_test.py::test_foo_1 PASSED                 [100%]

============== 2 passed, 2 deselected in 0.01s ==============

After reading the implementation of pytest_collection_modifyitems in the plugin, I've seen that this is how the plugin is implemented now, since it just loads a set of nodeids from the replay, but it is used just to filter away the unselected items.

Tweaking a bit the function I've managed to reproduce what I expected:

def pytest_collection_modifyitems(self, items, config):
    replay_file = config.getoption("replay_file")
    if not replay_file:
        return

    with open(replay_file, "r", encoding="UTF-8") as f:
        all_lines = f.readlines()
        nodeids = dict.fromkeys(
            json.loads(line)["nodeid"]
            for line in all_lines
            if not line.strip().startswith(("#", "//"))
        )

    remaining = []
    inds = set()
    for n in nodeids:
        ind = -1
        for i, item in enumerate(items):
            if item.nodeid == n:
                ind = i
                break
        if ind >= 0:
            remaining.append(items[i])
            inds.add(ind)

    deselected = [item for i, item in enumerate(items) if i not in inds]
    if deselected:
        config.hook.pytest_deselected(items=deselected)
    items[:] = remaining

The implementation is a bit raw since it was just a test.

What about adding this feature, maybe with it being optional and disabled by default? I can provide a PR if it's feasible to add as a feature.

[nicoddemus]

Hey @DavideCanton thanks for the detailed problem description!

Indeed it seems you are correct, it definitely seems like an oversight on our part -- the intention is to reproduce exactly what was executed, so the order of course matters.

Would you like to open a PR? I would be happy to review one.

[DavideCanton]

Sure, in the next days, I'll create a PR!

[nicoddemus]

Awesome, thanks!