Unify ways to run the Python test suite

[vstinner]

There are at least 15 known ways to run the Python test suite. IMO it's too much, it's hard to keep all these code paths consistent. IMO python -m test should be enough to fit all use cases.

Maybe we need a --ci, --buildbot, or --strict option which would run tests is "strict" mode.

---

Portable way to run tests:

./python -m test
./python -m test.regrtest
./python -m test.libregrtest
./python -m test.autotest

./python Lib/test/regrtest.py
./python Lib/test/autotest.py

./python -c 'from test import autotest'

The main drawback is that running ./python doesn't work when Python is built with ./configure --enable-shared, and it doesn't work when Python is cross-compiled (WASM/WASI buildbots). Moreover, on macOS, ./python.exe must be run instead. On Windows, it's just python.

Unix, from Python source code:

make buildbottest TESTOPTS="..."
make test
make testall
make testuniversal
make hostrunnertest

./python Tools/scripts/run_tests.py

Windows, from Python source code:

PCbuild\rt.bat
Tools\buildbot\test.bat

---

• Lib/test/autotest.py just runs test.libregrtest.main().

• Lib/test/regrtest.py runs test.libregrtest.main() and remove Lib/test/ from sys.path and make __file__ an absolute path.

• PCbuild\rt.bat:

  • python: pass -u -Wd -E -bb options
  • regrtest: pass options passed to rt.bat (except of options specific to rt.bat)
  • most options are to get the path to the python.exe program in PCbuild/

• Tools\buildbot\test.bat runs PCbuild\rt.bat:

  • rt.bat: pass -q -d options.
  • regrtest: pass -j1 -uall -rwW --slowest --timeout=1200 --fail-env-changed options.
  • ARM32 SSH pass -unetwork -udecimal -usubprocess -uurlfetch -utzdata -rwW --slowest --timeout=1200 --fail-env-changed options.

• make buildbottest runs Tools/scripts/run_tests.py:

  • python: pass $(TESTPYTHONOPTS) options.
  • regrtest: pass -j 1 -u all -W --slowest --fail-env-changed --timeout=$(TESTTIMEOUT) $(TESTOPTS) options.

• Tools/scripts/run_tests.py:

  • python: pass -u -W default -bb -E options -- also cross-compilation options, not detailed here
  • regrtest: pass -r -w -j0 -u all,-largefile,-audio,-gui options, add -n on Windows.

---

GitHub Action workflow uses, .github/workflows/build.yml:

• Windows x86: .\PCbuild\rt.bat -p Win32 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
• Windows x64: .\PCbuild\rt.bat -p x64 -d -q -uall -u-cpu -rwW --slowest --timeout=1200 -j0
• macOS: make buildbottest TESTOPTS="-j4 -uall,-cpu"
• Ubuntu: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"
• Address Sanitizer: xvfb-run make buildbottest TESTOPTS="-j4 -uall,-cpu"

Buildbot use, master/custom/factories.py:

• UnixBuild and UnixCrossBuild: make buildbottest TESTOPTS="..." TESTPYTHONOPTS="..." TESTTIMEOUT="..." with code to select TESTOPTS, TESTPYTHONOPTS and TESTTIMEOUT.
• UnixInstalledBuild: /path/to/python -m test ... with code to select -m test options.
• BaseWindowsBuild: Tools\buildbot\test.bat ... with code to select .... options

Linked PRs

• gh-109567
• gh-109570
• gh-109701
• gh-109909
• gh-109926
• gh-109954
• gh-109969
• gh-110062
• gh-110120
• gh-110121
• gh-110122

[vstinner]

Python options and regrtest options are now handled by new --fast-ci and --slow-ci options of regrtest.