pytest-repeat integration

[stas00]

Would it be possible to integrate pytest-repeat. That is if I have that and pytest-random-orderinstalled and I run:

pytest --count=10

it'll repeat the tests and shuffle the whole extended set. Currently the shuffle happens first and then the tests are repeated. What I'd like is for the shuffle to take the list of [tests * count] and to shuffle that.

Or perhaps it'd be just easier to just add repeat function to this module as it fits in quite closely.

Thank you.

[jbasko]

Perhaps I don't understand the issue, but with:

pytest=3.8.2
pytest-random-order=0.8.0
pytest-repeat=0.7.0
Python=3.6.6

it works with --count [n] as one would expect which is:

with default bucket module you will see tests and their repetitions shuffled within a module.

If you want to see tests shuffled globally then you should pass --random-order-bucket=global.

[stas00]

global is not what I am after. I need to stick to module buckets.

Say I have test modules: a.py b.py c.py

I'd like to be able to run it so that it will be something like:

pytest --count=2

a.py
c.py
b.py
c.py
a.py
b.py

I pretended that this was the randomization of course - any other order would do.

currently it runs:

c.py
a.py
b.py

repeating each module 2 times.

[jbasko]

Does this change do what you wish to see?

pip install git+https://github.com/jbasko/pytest-random-order#repeat

It's a bit hacky and I'm not sure what happens in different parametrize scenarios.

[stas00]

No, it still does the same.

Here is a quick way to reproduce it:

$ mkdir tests; cd tests
$ echo "def test_x(): assert True" | tee a.py | tee b.py | tee c.py
$ pytest --count=2 *py
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.6.6, pytest-3.8.2, py-1.5.4, pluggy-0.7.1
Using --random-order-bucket=module
Using --random-order-seed=38285

rootdir: /mnt/disc2/trash, inifile: setup.cfg
plugins: repeat-0.7.0, remotedata-0.3.0, random-order-0.8.0, pspec-0.0.3, openfiles-0.3.0, doctestplus-0.1.3, arraydiff-0.2
collected 6 items

c.py ..                                                                                                                                                              [ 33%]
a.py ..                                                                                                                                                              [ 66%]
b.py ..                                                                                                                                                              [100%]

===========================================

What I need is to randomize the order of files, so that the outcome will be similar to:

pytest --count=2 *py

a.py
c.py
b.py
c.py
a.py
b.py

Thank you.

[jbasko]

Apologies, the pip install command was wrong:

pip install git+https://github.com/jbasko/pytest-random-order.git@repeat

[stas00]

This is starting to head in the right direction, but not quite there yet, it now started randomizing globally:

a.py .                                                                                                                                                               [ 16%]
b.py ..                                                                                                                                                              [ 50%]
a.py .                                                                                                                                                               [ 66%]
c.py ..     

Please try my test case above - there should be 6 outputs at the end (3 files x count=2). You should be able to test on your side with that simple test case.

[jbasko]

It is not randomising globally. What it does is: 1) create buckets based on test item keys (which now include genid assigned by pytest-repeat as well as pytest.mark.parametrize (which pytest-repeat uses). 2) Shuffle items within each bucket. 3) Shuffle the order of buckets.

That is the overall algorithm of pytest-random-order and it won’t change. What can be done is new kinds of bucket keys can be created.

The idea of buckets is that items of certain origin end up in the same bucket which is guaranteed to be executed without overlapping (time wise) with items from another bucket.

Before I included genid you were observing that all tests from module a get duplicated and then shuffled among them. Now each repetition of module is a separate bucket.

[stas00]

I've been debugging this for a while, since your proposed version worked just fine with simple tests as in my example, but wasn't doing it right in the real test suite. I narrowed it down to @pytest.mark.parametrize - if a test file has it, it runs the files more than --count=count times when randomization is on.

Here is a test case to reproduce the problem:

echo -e "import pytest\n@pytest.mark.parametrize('a, b', [(1, 2), (3, 4), (5, 6)])\ndef test_x(a,b): assert True;assert True;" | tee a.py | tee b.py

pytest --count=2 *py
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.6.6, pytest-3.8.2, py-1.5.4, pluggy-0.7.1
Using --random-order-bucket=module
Using --random-order-seed=827761

rootdir: /mnt/disc2/trash, inifile: setup.cfg
plugins: repeat-0.7.0, remotedata-0.3.0, random-order-0.8.0, pspec-0.0.3, openfiles-0.3.0, doctestplus-0.1.3, arraydiff-0.2
collected 12 items

b.py ...                                                                                                                                                             [ 25%]
a.py ....                                                                                                                                                            [ 58%]
b.py .                                                                                                                                                               [ 66%]
a.py .                                                                                                                                                               [ 75%]
b.py .                                                                                                                                                               [ 83%]
a.py .                                                                                                                                                               [ 91%]
b.py .                               

as you can see it runs the files more than 2 times. So it splits up the tests from inside the file and I'd like to keep the tests intact inside each file and in the same order relative to each other. Only files to be randomized.

[jbasko]

I am closing this as I don't intend to change the behaviour.

As I see it, --count already works as expected as follows from the bucket idea. The purpose of bucket is to identify the boundaries of a set of tests which should not be interleaved with tests from other sets. If user wants to repeat tests in bucket X twice (in the example above in each module, but in particular in module a.py two times) then all the repetitions have to occur before or after tests from any other bucket. So n tests in a.py would produce 2*n consecutive tests from a.py.

The idea I explored in the other branch is following the same principle, but the repetition number (genid) is added to the bucket key so all tests from the first generation of a.py are in one bucket, and all tests from the second generation of a.py are in the second bucket. Now, because the design is to only ensure that one bucket is not interleaved with another, you could see all a-1's before a-2's or after, and they could have b-*'s in between.

For inspection of test order I suggest using --collect-only.

[Mrodent]

I think there is a need for this: I have taken to going

for i in {1..10}; do pytest --random-order; done

when I suspect there's something amiss and intermittent (which happens quite a lot, particularly as I'm using pytest-qt).

It would be nice if that could just be a quick switch...

Here's a bash script: NB link to pytest-random-order

#!/bin/bash

<<comment
2022-02-04
This is a script to run pytest multiple times... the switch you enter is -c followed by a number
Other than that, just enter arguments as normal.
Example:
./pyt.sh -s -c 5 -k my_test_file
... the -c switch can go anywhere. 
NB the arg "--random-order" is added automatically: if you're not doing
a random order run there's likely to be no need to repeat the run.

comment

POSITIONAL_ARGS=()
TIMES_TO_RUN=1

while [[ $# -gt 0 ]]; do
  case $1 in
    -c)
      TIMES_TO_RUN="$2"
      shift
      shift
      ;;  
    *)
      POSITIONAL_ARGS+=("$1") # save positional arg
      shift # past argument
      ;;
  esac
done
set -- "${POSITIONAL_ARGS[@]}" # restore positional parameters
echo "TIMES_TO_RUN  = ${TIMES_TO_RUN}"
for i in $(seq 1 $TIMES_TO_RUN); do
  pytest --random-order $*
done