Open willingc opened 1 year ago
This makes sense.
The linked "Run the tests" gives more info:
If you have a multi-core or multi-CPU machine, you can enable parallel testing using several Python processes so as to speed up things:
./python -m test -j0
If you are running a version of Python prior to 3.3 you must specify the number of processes to run simultaneously (e.g.
-j2
).
Let's recommend -j0
on the landing page, and we could add more info for older/single-core machines to this longer description.
And it's probably worth removing the "prior to 3.3" by now, 3.2 is EOL since 2016.
@hugovk I would have some mention of older/single core on the landing page since in some parts of the world older/single core machines are the norm. It can refer them to "Run the tests" link for more info.
It is worth updating the GA of CPython too.
Sorry for my ignorance but how does -j0 differ from -j?
From the GNU docs :
If the ‘-j’ option is followed by an integer, this is the number of recipes to execute at once; this is called the number of job slots. If there is nothing looking like an integer after the ‘-j’ option, there is no limit on the number of job slots.
I just wondered why step 3 for build python uses -j and the recommendation for tests is -j0. What's the difference between them?
I'd be happy to have a go at this as a first contribution but just wanted to check my understanding.
There's some inconsistency in the -j
options for make
and for ./python -m test
.
On macOS:
❯ make -j8
Checked 107 modules (31 built-in, 76 shared, 0 n/a on macosx-13.6-arm64, 0 disabled, 0 missing, 0 failed on import)
❯ make -j
Checked 107 modules (31 built-in, 76 shared, 0 n/a on macosx-13.6-arm64, 0 disabled, 0 missing, 0 failed on import)
❯ make -j0
make: the `-j' option requires a positive integral argument
Usage: make [options] [target] ...
Options:
-b, -m Ignored for compatibility.
-B, --always-make Unconditionally make all targets.
-C DIRECTORY, --directory=DIRECTORY
Change to DIRECTORY before doing anything.
-d Print lots of debugging information.
--debug[=FLAGS] Print various types of debugging information.
-e, --environment-overrides
Environment variables override makefiles.
-f FILE, --file=FILE, --makefile=FILE
Read FILE as a makefile.
-h, --help Print this message and exit.
-i, --ignore-errors Ignore errors from commands.
-I DIRECTORY, --include-dir=DIRECTORY
Search DIRECTORY for included makefiles.
-j [N], --jobs[=N] Allow N jobs at once; infinite jobs with no arg.
-k, --keep-going Keep going when some targets can't be made.
-l [N], --load-average[=N], --max-load[=N]
Don't start multiple jobs unless load is below N.
-L, --check-symlink-times Use the latest mtime between symlinks and target.
-n, --just-print, --dry-run, --recon
Don't actually run any commands; just print them.
-o FILE, --old-file=FILE, --assume-old=FILE
Consider FILE to be very old and don't remake it.
-p, --print-data-base Print make's internal database.
-q, --question Run no commands; exit status says if up to date.
-r, --no-builtin-rules Disable the built-in implicit rules.
-R, --no-builtin-variables Disable the built-in variable settings.
-s, --silent, --quiet Don't echo commands.
-S, --no-keep-going, --stop
Turns off -k.
-t, --touch Touch targets instead of remaking them.
-v, --version Print the version number of make and exit.
-w, --print-directory Print the current directory.
--no-print-directory Turn off -w, even if it was turned on implicitly.
-W FILE, --what-if=FILE, --new-file=FILE, --assume-new=FILE
Consider FILE to be infinitely new.
--warn-undefined-variables Warn when an undefined variable is referenced.
-N OPTION, --NeXT-option=OPTION
Turn on value of NeXT OPTION.
This program built for i386-apple-darwin11.3.0
Report bugs to <bug-make@gnu.org>
And:
❯ ./python.exe -m test -j8
== CPython 3.13.0a0 (heads/main:6ab6040054e5, Oct 5 2023, 08:37:40) [Clang 15.0.0 (clang-1500.0.40.1)]
== macOS-13.6-arm64-arm-64bit little-endian
== Python build: debug
== cwd: /Users/hugo/github/cpython/build/test_python_worker_12564æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
== resources: (all disabled, use -u option)
...
❯ ./python.exe -m test -j0
== CPython 3.13.0a0 (heads/main:6ab6040054e5, Oct 5 2023, 08:37:40) [Clang 15.0.0 (clang-1500.0.40.1)]
== macOS-13.6-arm64-arm-64bit little-endian
== Python build: debug
== cwd: /Users/hugo/github/cpython/build/test_python_worker_12634æ
== CPU count: 8
== encodings: locale=UTF-8, FS=utf-8
== resources: (all disabled, use -u option)
...
❯ ./python.exe -m test -j
usage: python -m test [options] [test_name1 [test_name2 ...]]
python path/to/Lib/test/regrtest.py [options] [test_name1 [test_name2 ...]]
regrtest.py: error: argument -j/--multiprocess: expected one argument
Pass -h or --help for complete help.
make
does not necessarily imply GNU Make; our makefile can also be run with BSD Make. My bsdmake
only accepts -j
with a positive integer argument.
GNU Make:
-j [jobs], --jobs[=jobs]
Specifies the number of jobs (commands) to run simultaneously. If
there is more than one -j option, the last one is effective. If
the -j option is given without an argument, make will not limit
the number of jobs that can run simultaneously.
BSD Make:
-j max_jobs
Specify the maximum number of jobs that bsdmake may have running
at any one time. Turns compatibility mode off, unless the B flag
is also specified.
So -j0
is the safe option for both these make
s and python -m test
.
Separately, we can't easily change all these make
flavours, but we could make python -m test
more accepting and add python -m test -j
to do the same as python -m test -j0
?
Thanks for the info
Hey, if the decision has been taken as specified by @willingc I would kindly love to work on this issue meanwhile let me be working on some other issue as I wait for this. Thanks
On the landing page, we recommend running:
./python -m test -j3
Since most systems now are multicore, we should consider recommending:
./python -m test -j0
Tests on my machine several minutes faster. We could continue to recommend./python -m test -j3
for folks using older or single-core machines.