Closed egorenar closed 1 year ago
For some reason, OpenWrt is using staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py
instead of build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/Python-3.11.5/Lib/distutils/command/build_ext.py
to build lxml.builder extension module.
Another hint:
$ rg "LIBDIR'.*/usr/lib" staging_dir
staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py
708: 'LIBDIR': '/usr/lib',
Very similar to this https://github.com/openwrt/packages/issues/9275.
For some reason build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/Python-3.11.5/Lib/distutils/sysconfig.py
change from patch lang/python/python3/patches/008-distutils-use-python-sysroot.patch
has either been reverted or never applied.
I'm able to build python-lxml with no issues. Can you try building with a clean buildroot/sdk?
I did a clean build with master and LIBDIR
is still /usr/lib
in staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py
.
If i change LIBDIR
by hand to ... staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib
then everything compiles.
Can you post a complete build log?
Edit: also what OS/version is your build host?
Edit 2: please include the build log for python as well as python-lxml
Can you post a complete build log?
Edit: also what OS/version is your build host?
Edit 2: please include the build log for python as well as python-lxml
Linux distro: Void Linux (native Python 3.11 installed in /usr/lib !!!) Build log for python3 and python-lxml attached:
make V=sc -j$(($(nproc)+1)) package/feeds/packages/python3/{clean,compile} 2>&1 | tee build.log
make V=sc -j$(($(nproc)+1)) package/python-lxml/{clean,compile} 2>&1 | tee -a build.log
thanks for looking into it.
Do you get the same error if you compile for an existing arch/target instead of your custom target?
Built the target ipq806x
model Nighthawk X4S R7800
from master 359a6e36e972 ("mac80211: ath11k: sync with ath-next")
. Enabled python3-lxml
as the only extra packaged. The problem persists.
Build instructions:
$ git clone https://github.com/openwrt/openwrt.git openwrt-test
$ cd openwrt-test
$ make menuconfig
# Target: ipq806x
# Model: Nighthawk X4S R7800
$ ./scripts/feeds update -a && ./scripts/feeds install -a
$ make menuconfig
# select python3-lxml
$ grep lxml .config
CONFIG_PACKAGE_python3-lxml=y
$ make V=s -j$(($(nproc)+1)) 2>&1 | tee build.log
$ grep '\bLIBDIR\b' staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py
'LIBDIR': '/usr/lib',
$ rg "sysconfig.get_config_var.*LIBDIR" staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi
staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/distutils/command/build_ext.py
237: libdir = sysconfig.get_config_var('LIBDIR')
Oh forgot about another interesting fact :) Void Linux updated Python 3 to version 3.12 recently and since then everything builds again, lxml as well, no failures. I think the reason is because OpenWrt builds Python 3.11 which is no longer present on Void Linux, and, therefore, OpenWrt fails to link the native Void Linux library.
But still, the problem is that LIBDIR is pointing to /usr/lib.
I found something out.
python3-lxml uses staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py
to compile C extensions.
Made the following change:
--- staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py.orig 2023-10-14 14:30:47.564996239 +0200
+++ staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py 2023-10-14 14:28:43.620002109 +0200
@@ -240,6 +240,9 @@
def build_extension(self, ext):
ext._convert_pyx_sources_to_lang()
_compiler = self.compiler
+
+ log.warn("--- DEBUG1 --: {}".format(self.library_dirs))
+
try:
if isinstance(ext, Library):
self.compiler = self.shlib_compiler
And rebuilding the package produces this output:
...
--- DEBUG1 --: ['/usr/lib']
...
I still do not understand it properly, but there is no "_python_sysroot" support in this build_ext.py as described here: https://github.com/openwrt/packages/pull/9288
Next finding :)
When lxml is being built then staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py
inherits from staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py
.
And the latter also doesn't support _python_sysroot
.
if sysconfig.get_config_var('Py_ENABLE_SHARED'):
if not sysconfig.python_build:
# building third party extensions
self.library_dirs.append(sysconfig.get_config_var('LIBDIR'))
else:
# building python standard extensions
self.library_dirs.append('.')
No "_python_sysroot" :/
Do we need to patch setuptool's build_ext.py ? As it is done here: https://github.com/openwrt/packages/blob/master/lang/python/python3/patches/008-distutils-use-python-sysroot.patch
I applied this patch https://github.com/openwrt/packages/blob/53dc7146f4623a99b62da5918b3f4ce002697a32/lang/python/python3/patches/008-distutils-use-python-sysroot.patch#L30C18-L33C50
to staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py
and i no longer see -L/usr/lib
. Instead i see -L/home/egorenar/Repositories/openwrt-test/staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib
The buildbots have no problem building this package. I am using Ubuntu 23.04 with Python 3.11.4 installed and I have no problem building this package.
I'd like to understand what is different about your setup that is causing you problems before applying any solutions.
Are you using Void Linux with glibc or musl?
Are you using Void Linux with glibc or musl?
glibc
So the reason why the build succeeds on Debian/Ubuntu (and I believe Fedora) is because:
STAGING_DIR
is patched into the GCC specsIt is failing for you because libpython3.11.so is in /usr/lib on Void and the -L/usr/lib
is ahead of the path patched into the GCC specs.
The fix is in https://github.com/openwrt/packages/pull/22551 - thanks for your help in diagnosing the issue and for your patience :pray:
thanks for looking into this!
Maintainer: @commodo Environment: ARMv7 Processor rev 4 (v7l), OpenWrt SNAPSHOT r24099-5dfd8dc7d4f5
python3-lxml fails to build because it tries to link host's Python3 library. The compiler is using
-L/usr/lib
.