YoSTEALTH
commented
3 months ago Any comment on this?
Closed YoSTEALTH closed 3 months ago
YoSTEALTH
commented
3 months ago Any comment on this?
layday
commented
3 months ago build_ext is passed to both the build_sdist and build_wheel hooks so it ends up being invoked twice, I imagine. There was a proposal to split config settings by hook but it was rejected. I don't know if there's some other way you can configure setuptools to work around this issue.
henryiii
commented
3 months ago I think this is referring to building dependencies in build-system.requires. This should be rare except on systems without many or any wheels.
henryiii
commented
3 months ago Ahh, no, didn’t read it properly.
henryiii
commented
3 months ago This really needs to be fixed in setuptools. Other build backends (like scikit-build-core and meson-python) don’t have this problem.
henryiii
commented
3 months ago And you can work around it by reading environment variables in setup.py. Setuptools’ configuration system was designed around setup.py command and hasn’t been adapted to work with modern building very well. And setuptools doesn’t support compiling files in parallel in the first place, so it either is already hacked in or you are only getting extensions building in parallel. Actually, it looks like you have one extension (so this won’t affect anything) and are already running “make” in parallel: https://github.com/YoSTEALTH/Liburing/blob/7c584e78d40fe6b0552af79085ba27f0901e8831/setup.py#L12 ? Have you checked to see if passing this is faster? I’d guess it might not be.
YoSTEALTH
commented
3 months ago @henryiii That threads variable is for make and cython which is done very fast. It does not effect the build_ext setuptools uses.
Looks like this again is setuptools issue, I will try and make a post there. The reason I posted here is because build --help says:
By default, a source distribution (sdist) is built from {srcdir} and a binary distribution (wheel) is built from the sdist. This is recommended as it will ensure the sdist can be used to build wheels. Figured
wheelwas being built bysdist
I am stuck with using setuptools for this release, maybe later I can look into other build options if thing don't get better.
Thanks @layday and @henryiii for your comments.
henryiii
commented
3 months ago build_ext does not build an Extension's files in parallel. That switch only allows Extensions to be build in parallel. You have one Extension so it should have no effect.
YoSTEALTH
commented
3 months ago @henryiii I see, makes sense.
All I know is https://github.com/YoSTEALTH/Liburing/tree/master/src/liburing each of those .pxy will create its own .so file. build_ext by default will compile 1 at a time, running build_ext -j4 will build multiple of those.
henryiii
commented
3 months ago Ah, so cythonize returns a list of Extensions: https://github.com/cython/cython/blob/6e8585b3def3a5b8a6ab9a3291015a8806024b6a/Cython/Build/Dependencies.py#L851 - yes, multiple extensions do compile in parallel.
You should be able to customize built_ext and set self.parallel inside build_extensions, I think. I've never tried setting it, only reading it, but I think it might/should work.
YoSTEALTH
commented
3 months ago Just checked cythonize(nthreads=threads) is needed as it will convert *.pxy to *.c in parallel even in single Extensions.
So I ended up writing a subclass according to your recommendation, this solves the build_ext, build, and pip parallel issues...
It still runs the ./configure couple of times, I think I should move it into the subclass as well (might solve the issue as well).
Thank you @henryiii for being awesome and helping out https://github.com/YoSTEALTH/Liburing/commit/520fe48784e03679aa547e7516181c5ffbe269af
abravalheri
commented
3 months ago For future reference. This is the resolution for the issue provided in https://github.com/pypa/setuptools/issues/4290:
Please note that the command python3 -m build -C--global-option="build_ext -j16" is "sneaking in" build_ext -j16 to every single build hook API (these hooks were initially defined in PEP 517/660) and that is the reason why the build runs multiple times. Note that in the given example, this behaviour is explicitly solicited by the user input via CLI arguments.
If you don't want that behaviour please do not use -C--global-option="build_ext -j16".
Note that --global-option is provided as it is: an experimental attempt to emulate pip's old behaviour, but it is not guaranteed to do exactly what you want (specially because pip's old behaviour is not 100% equivalent/compatible with the build process initially introduced in PEP 517/660).
If you want to pass parameters to the build_ext, the only guaranteed way that works right now is to use setup.cfg to set the parallel option. See https://setuptools.pypa.io/en/latest/deprecated/distutils/configfile.html for reference.
For project Liburing when I run
python3 -m build -C--global-option="build_ext -j16"looks like build compiles the code multiple times!Ideally it would be nice if
buildcompiled once and used the same files for both--sdistand--wheelMaybe there is a underlying relation to https://github.com/pypa/pip/issues/12604 issue.