Running pytest-repeat together with pytest-randomly does not give new random seed

[Cecron]

Would it be possible to run pytest-repeat together with pytest-randomly and get a new random seed for each session?

Trying to run with --repeat-scope=session, did unfortunately not trigger a new randomly-seed.

$ pytest tests/hdl/test_signal_decoding_tb.py::test_fpga_random_transactions -lsx --repeat-scope=session --count=2
================================ test session starts =================================
platform linux -- Python 3.8.10, pytest-7.1.2, pluggy-1.0.0
Using --randomly-seed=2806041905
rootdir: /home/user
plugins: randomly-3.12.0, repeat-0.9.3, xdist-3.1.0, cocotb-test-0.2.2
collected 2 items

tests/hdl/test_signal_decoding_tb.py::test_fpga_random_transactions[1-2] PASSED
tests/hdl/test_signal_decoding_tb.py::test_fpga_random_transactions[2-2] PASSED

================================= 2 passed in 0.02s ==================================

Thanks for a great plugin!

[okken]

It would not, the way pytest-randomly is written now.

pytest-randomly

• generates the seed at import time, https://github.com/pytest-dev/pytest-randomly/blob/884b80330f695d56d475ad5f3b192d2bf3a5e381/src/pytest_randomly/__init__.py#L71
• or grabs a passed in seed at config time, https://github.com/pytest-dev/pytest-randomly/blob/884b80330f695d56d475ad5f3b192d2bf3a5e381/src/pytest_randomly/__init__.py#L116

pytest-repeat

• generates all repetitions at the time of pytest_generate_tests, https://github.com/pytest-dev/pytest-repeat/blob/b48328df0a481c4393121e3d51f2147ca18f47d9/pytest_repeat.py#L55C1-L55C37
• even repeat-scope=session still only generates the list once, and that happens after pytest_config, and most definitely after pytest-randomly is imported.

I'm not saying this is impossible. But pytest-randomly and pytest-repeat would both have to change such that at least one knows about the other and can detect when to update the seed, or record the seed between session loops, or something. That's rather unlikely, I'm afraid

Then

[Cecron]

I was afraid of that. ;-)

If anyone else is looking for this, my current workaround is to run the test from bash, e.g. the script below runs for a specific time (kind of like pytest-flakefinder):

#!/bin/bash

counter=0  # The counter is for information
timeout_seconds=10  # Adjust the timeout duration as needed

while true; do
    ((counter++))
    # Run the test with a timeout
    timeout $timeout_seconds pytest -o log_cli=0 -o log_cli_level=DEBUG tests/hdl/test_signal_decoding_tb.py::test_fpga_random_transactions -lsx
    exit_code=$?
    # Check the exit code to determine success or failure
    if [ $exit_code -eq 124 ]; then
        echo "Test timed out. Stopping the loop."
        break
    elif [ $exit_code -ne 0 ]; then
        echo "Test failed with exit code $exit_code. Stopping the loop."
        break
    else
        echo "Test completed successfully."
    fi
    # sleep 1  # Optionally add a delay before the next iteration
done

Thanks for the explanation!