pytest-dev / pytest-random-order

pytest plugin to randomise the order of tests with some control over the randomness
MIT License
67 stars 10 forks source link

Multiple runs? #47

Closed Mrodent closed 9 months ago

Mrodent commented 2 years ago

I find myself wanting to do multiple runs for maximum robustness. Maybe this could be added. Here's a bash script I use:

#!/bin/bash

<<comment
2022-02-04
This is a script to run pytest multiple times... the switch you enter is -c followed by a number
Other than that, just enter arguments as normal.

Example:

./pyt.sh -s -c 5 -k my_test_file

... the -c switch can go anywhere
NB the arg "--random-order" is added automatically, as this is the reason you'd want to do multiple runs
comment

POSITIONAL_ARGS=()
TIMES_TO_RUN=1

while [[ $# -gt 0 ]]; do
  case $1 in
    -c)
      TIMES_TO_RUN="$2"
      shift
      shift
      ;;  

    *)
      POSITIONAL_ARGS+=("$1") # save positional arg
      shift # past argument
      ;;
  esac
done

set -- "${POSITIONAL_ARGS[@]}" # restore positional parameters

echo "Running ${TIMES_TO_RUN} times"

for i in $(seq 1 $TIMES_TO_RUN); do
  pytest --random-order $*
done
jbasko commented 9 months ago

Closing this as there are other plugins for this, for example, https://github.com/pytest-dev/pytest-repeat