python3-lxml: trying to link host python3 library

[egorenar]

Maintainer: @commodo Environment: ARMv7 Processor rev 4 (v7l), OpenWrt SNAPSHOT r24099-5dfd8dc7d4f5

python3-lxml fails to build because it tries to link host's Python3 library. The compiler is using -L/usr/lib.

make V=sc -j$(($(nproc)+1)) package/feeds/packages/python-lxml/{clean,compile}
...
arm-openwrt-linux-muslgnueabi-gcc -shared -L/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/usr/lib -L/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/lib -fuse-ld=bfd -znow -zrelro -lpython3.11 -Os -pipe -fno-caller-saves -fno-plt -fhonour-copts -mfloat-abi=hard -fmacro-prefix-map=/home/egorenar/Repositories/openwrt-rel/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/pypi/lxml-4.9.3=lxml-4.9.3 -Wformat -Werror=format-security -fstack-protector -D_FORTIFY_SOURCE=1 -Wl,-z,now -Wl,-z,relro -I/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/usr/include -I/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/include/fortify -I/home/egorenar/Repositories/openwrt-rel/staging_dir/toolchain-arm_cortex-a15+neon-vfpv4_gcc-12.3.0_musl_eabi/include -I/home/egorenar/Repositories/openwrt-rel/staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/include/python3.11 build/temp.linux-arm-cpython-311/src/lxml/builder.o -L/usr/lib -o build/lib.linux-arm-cpython-311/lxml/builder.cpython-311-arm-linux-musleabihf.so
/usr/lib/libpython3.11.so: file not recognized: file format not recognized
collect2: error: ld returned 1 exit status

[egorenar]

For some reason, OpenWrt is using staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py instead of build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/Python-3.11.5/Lib/distutils/command/build_ext.py to build lxml.builder extension module.

[egorenar]

Another hint:

$ rg "LIBDIR'.*/usr/lib" staging_dir
staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py
708: 'LIBDIR': '/usr/lib',

Very similar to this https://github.com/openwrt/packages/issues/9275.

[egorenar]

For some reason build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/Python-3.11.5/Lib/distutils/sysconfig.py change from patch lang/python/python3/patches/008-distutils-use-python-sysroot.patch has either been reverted or never applied.

[jefferyto]

I'm able to build python-lxml with no issues. Can you try building with a clean buildroot/sdk?

[egorenar]

I did a clean build with master and LIBDIR is still /usr/lib in staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py.

If i change LIBDIR by hand to ... staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib then everything compiles.

[jefferyto]

Can you post a complete build log?

Edit: also what OS/version is your build host?

Edit 2: please include the build log for python as well as python-lxml

[egorenar]

Can you post a complete build log?

Edit: also what OS/version is your build host?

Edit 2: please include the build log for python as well as python-lxml

Linux distro: Void Linux (native Python 3.11 installed in /usr/lib !!!) Build log for python3 and python-lxml attached:

make V=sc -j$(($(nproc)+1)) package/feeds/packages/python3/{clean,compile} 2>&1 | tee build.log
make V=sc -j$(($(nproc)+1)) package/python-lxml/{clean,compile} 2>&1 | tee -a build.log

build.log.gz

thanks for looking into it.

[jefferyto]

Do you get the same error if you compile for an existing arch/target instead of your custom target?

[egorenar]

Built the target ipq806x model Nighthawk X4S R7800 from master 359a6e36e972 ("mac80211: ath11k: sync with ath-next"). Enabled python3-lxml as the only extra packaged. The problem persists.

Build instructions:

$ git clone https://github.com/openwrt/openwrt.git openwrt-test
$ cd openwrt-test
$ make menuconfig
# Target: ipq806x
# Model: Nighthawk X4S R7800
$ ./scripts/feeds update -a && ./scripts/feeds install -a
$ make menuconfig
# select python3-lxml
$ grep lxml .config
CONFIG_PACKAGE_python3-lxml=y
$ make V=s -j$(($(nproc)+1)) 2>&1 | tee build.log

$ grep '\bLIBDIR\b' staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/_sysconfigdata__linux_arm-linux-musleabihf.py
 'LIBDIR': '/usr/lib',

$ rg "sysconfig.get_config_var.*LIBDIR" staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi
staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib/python3.11/distutils/command/build_ext.py
237:                libdir = sysconfig.get_config_var('LIBDIR')

build.log.gz

[egorenar]

Oh forgot about another interesting fact :) Void Linux updated Python 3 to version 3.12 recently and since then everything builds again, lxml as well, no failures. I think the reason is because OpenWrt builds Python 3.11 which is no longer present on Void Linux, and, therefore, OpenWrt fails to link the native Void Linux library.

But still, the problem is that LIBDIR is pointing to /usr/lib.

[egorenar]

I found something out. python3-lxml uses staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py to compile C extensions.

Made the following change:

--- staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py.orig       2023-10-14 14:30:47.564996239 +0200
+++ staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py    2023-10-14 14:28:43.620002109 +0200
@@ -240,6 +240,9 @@
     def build_extension(self, ext):
         ext._convert_pyx_sources_to_lang()
         _compiler = self.compiler
+
+        log.warn("--- DEBUG1 --: {}".format(self.library_dirs))
+
         try:
             if isinstance(ext, Library):
                 self.compiler = self.shlib_compiler

And rebuilding the package produces this output:

...
--- DEBUG1 --: ['/usr/lib']
...

I still do not understand it properly, but there is no "_python_sysroot" support in this build_ext.py as described here: https://github.com/openwrt/packages/pull/9288

[egorenar]

Next finding :) When lxml is being built then staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/command/build_ext.py inherits from staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py. And the latter also doesn't support _python_sysroot.

        if sysconfig.get_config_var('Py_ENABLE_SHARED'):
            if not sysconfig.python_build:
                # building third party extensions
                self.library_dirs.append(sysconfig.get_config_var('LIBDIR'))
            else:
                # building python standard extensions
                self.library_dirs.append('.')

No "_python_sysroot" :/

[egorenar]

Do we need to patch setuptool's build_ext.py ? As it is done here: https://github.com/openwrt/packages/blob/master/lang/python/python3/patches/008-distutils-use-python-sysroot.patch

[egorenar]

I applied this patch https://github.com/openwrt/packages/blob/53dc7146f4623a99b62da5918b3f4ce002697a32/lang/python/python3/patches/008-distutils-use-python-sysroot.patch#L30C18-L33C50 to staging_dir/hostpkg/lib/python3.11/site-packages/setuptools/_distutils/command/build_ext.py and i no longer see -L/usr/lib. Instead i see -L/home/egorenar/Repositories/openwrt-test/staging_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/usr/lib

[jefferyto]

The buildbots have no problem building this package. I am using Ubuntu 23.04 with Python 3.11.4 installed and I have no problem building this package.

I'd like to understand what is different about your setup that is causing you problems before applying any solutions.

[jefferyto]

Are you using Void Linux with glibc or musl?

[egorenar]

Are you using Void Linux with glibc or musl?

glibc

[jefferyto]

So the reason why the build succeeds on Debian/Ubuntu (and I believe Fedora) is because:

• There is no libpython3.11.so in /usr/lib (Debian/Ubuntu has multiarch directories and libpython3.11.so is in /usr/lib/x86_64-linux-gnu, I believe on Fedora the shared library is in /usr/lib64)
• The linker still finds the correct shared library because STAGING_DIR is patched into the GCC specs

It is failing for you because libpython3.11.so is in /usr/lib on Void and the -L/usr/lib is ahead of the path patched into the GCC specs.

The fix is in https://github.com/openwrt/packages/pull/22551 - thanks for your help in diagnosing the issue and for your patience :pray:

[egorenar]

thanks for looking into this!