Closed YoSTEALTH closed 1 month ago
Please note that the command python3 -m build -C--global-option="build_ext -j16"
is "sneaking in" build_ext -j16
to every single build hook API (these hooks were initially defined in PEP 517/660) and that is the reason why the build runs multiple times. Note that in the given example, this behaviour is explicitly solicited by the user input via CLI arguments.
If you don't want that behaviour please do not use -C--global-option="build_ext -j16"
.
Note that --global-option
is provided as it is: an experimental attempt to emulate pip
's old behaviour, but it is not guaranteed to do exactly what you want (specially because pip
's old behaviour is not 100% equivalent/compatible with the build process initially introduced in PEP 517/660).
If you want to pass parameters to the build_ext
, the only guaranteed way that works right now is to use setup.cfg
to set the parallel
option. See https://setuptools.pypa.io/en/latest/deprecated/distutils/configfile.html for reference.
@abravalheri To clarify, build_ext
will be invoked in all cases, the user will simply need to define any custom args in setup.cfg
?
To clarify,
build_ext
will be invoked in all cases, the user will simply need to define any custom args insetup.cfg
?
By default, build_ext
is automatically called when creating the wheel (as far as I remember). And then it pulls configuration parameters from setup.cfg
.
When using the CLI in the same way exemplified in the original text of the issue, build_ext
is called multiple times.
Thanks for explaining!
Sent with Proton Mail secure email.
On Friday, 5 April 2024 at 17:54, Anderson Bravalheri @.***> wrote:
To clarify, build_ext will be invoked in all cases, the user will simply need to define any custom args in setup.cfg?
By default, build_ext is called when creating the wheel (as far as I remember). And then it pulls configuration parameters from setup.cfg.
When using the CLI in the same way exemplified in the original text of the issue, build_ext is called multiple times.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
This issue had nothing to do with using python3 -m build -C--global-option="build_ext -j16"
I just added it there so you don't have to wait ages to see the output.
Anyways by default if you ran python3 -m pip install --upgrade git+https://github.com/YoSTEALTH/Shakti -v
you would still see multiple compiles. I found a fix to this problem by sub-classing from setuptools.command.build import build
:
class Build(build):
def initialize_options(self):
super().initialize_options()
self.parallel = threads # manually set
edit: For people that didn't bother to subclass setup will still run multiple compiles, so the problem still exists.
Are we talking about the compilation of the C code multiple times or the cythonize
?
The cythonize
does run multiple times, because we need to run the setup.py
every time a build hook is called. The cythonize
function is eager and I don't remember if it checks for cached files.
But as far as I remember the compilation of the C code should happen only when the wheel is built (unless there is heavy customisation or if you add build_ext
in the CLI).
For lazy cythonization, I believe you can create the Extension
objects directly with mentioning the .pyx
files in the source argument, without using the cythonize
function. Not sure if vanilla setuptools
supports glob pattern there though.
No, there is nothing wrong with cythonize(nthreads=threads)
as it convert .pxy to .c in parallel even in single Extensions.
The problem was with https://github.com/YoSTEALTH/Liburing/tree/master/src/liburing each of those .pxy will create its own .so file, and by default setuptools will create all those in single thread, so using -j4
or self.parallel=True
was important.
Without sub-classing it took ages for compile and setuptool would compile again!
I can revert the changes I made to the site if you want to see the output again.
@abravalheri ok, go ahead and run python3 -m pip install --upgrade git+https://github.com/YoSTEALTH/Shakti -vvv
to see the output
I feel like I am loosing my mind running these compiles, now it doesn't seem to run compile multiple times even though it runs ./configure
multiple times. With the past of trace-back above you can see it running multiple times.
same with build_ext -j4
or without it.
It might be a good idea for setuptools
to set as default
self.parallel = True
# or
self.parallel = os.cpu_count()//2 or 1 # use half of cpu resources
This might help a lots of library out there to install/build/... faster.
ok, go ahead and run
python3 -m pip install --upgrade git+https://github.com/YoSTEALTH/Shakti -vvv
to see the output
Yes, please find the logs below:
We can see here that it compiles the C code for Shakti
only once when creating the wheel.
It does compile the code for liburing
once for the build environment and another one for the installation environment. This is due to the combined effect of the build isolation (opt-out behaviour in pip
when you have pyproject.toml
) + the limitation of pip
when it comes to caching the wheel from VCS reference. But this part was already explained in https://github.com/pypa/pip/issues/12604#issuecomment-2029705978, so I assume that is not the reason why you opened this issue in setuptools.
setuptools version
69.2.0
Python version
3.13-dev
OS
Arch Linux 6.7+
Additional environment information
No response
Description
For project Liburing when I run python3 -m build -C--global-option="build_ext -j16" looks like build compiles the code multiple times!
Already posted in:
Looks like this is
setuptools
issue.Expected behavior
build-system.requires
andproject.dependencies
How to Reproduce
python3 -m pip install --upgrade --config-setting="--build-option=build_ext -j4" git+https://github.com/YoSTEALTH/Liburing
python3 -m pip install --upgrade --config-setting="--build-option=build_ext -j4" git+https://github.com/YoSTEALTH/Shakti -vvv
Output