search

openwrt / packages

Community maintained packages for OpenWrt. Documentation for submitting pull requests is in CONTRIBUTING.md
GNU General Public License v2.0
4.01k stars 3.48k forks source link

libgpiod: does not compile on macOS 11.2-x86_64 host #14652

Closed k-ronny closed 3 years ago

k-ronny commented 3 years ago

Maintainer: @mhei Environment: all?

Description: cross compiling of libgpiod on a macOS 11.2-x86_64 host is failing

...
gpiodmodule.c:8:10: fatal error: Python.h: No such file or directory
 #include <Python.h>
          ^~~~~~~~~~
compilation terminated.
...

I think it is related to this definition in the Makefile:

CONFIGURE_VARS += \
        PYTHON="$(STAGING_DIR_HOSTPKG)/bin/$(PYTHON3)" \
        PYTHON_CPPFLAGS="$(shell $(STAGING_DIR)/host/bin/$(PYTHON3)-config --includes)" \
        PYTHON_LIBS="$(shell $(STAGING_DIR)/host/bin/$(PYTHON3)-config --libs)"
endif

If I apply this patch it compiles on macOS and Debian Buster and it runs on a Onion Omega2+:

diff --git a/libs/libgpiod/Makefile b/libs/libgpiod/Makefile
index 68996c831..b944dbcfa 100644
--- a/libs/libgpiod/Makefile
+++ b/libs/libgpiod/Makefile
@@ -33,9 +33,8 @@ endif
 ifneq ($(CONFIG_PACKAGE_python3-gpiod),)
 CONFIGURE_ARGS += --enable-bindings-python
 CONFIGURE_VARS += \
-       PYTHON="$(STAGING_DIR_HOSTPKG)/bin/$(PYTHON3)" \
-       PYTHON_CPPFLAGS="$(shell $(STAGING_DIR)/host/bin/$(PYTHON3)-config --includes)" \
-       PYTHON_LIBS="$(shell $(STAGING_DIR)/host/bin/$(PYTHON3)-config --libs)"
+       PYTHON_CPPFLAGS="-I$(PYTHON3_INC_DIR)" \
+       PYTHON_LIBS="-lpython$(PYTHON3_VERSION)"
 endif

 define Package/libgpiod

But I do not know if this is the right approach.

neheb commented 3 years ago

ping @jefferyto

neheb commented 3 years ago

@k-ronny what does $(STAGING_DIR)/host/bin/$(PYTHON3)-config --includes actually return?

k-ronny commented 3 years ago

@neheb

k_ronny@pollux openwrt/ > staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
zsh: staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config: bad interpreter: /usr/bin/python3.9: no such file or directory

this is because

k_ronny@pollux openwrt/ > head -1 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config 
#!/usr/bin/python3.9

and in my build environment python3 is

k_ronny@pollux openwrt/ > type python3
python3 is /usr/local/bin/python3

this results in

<...> PYTHON="/Volumes/work/openwrt/staging_dir/hostpkg/bin/python3.9" PYTHON_CPPFLAGS="" PYTHON_LIBS="" <...>

and later to

gmake[7]: Entering directory '/Volumes/work/openwrt/build_dir/target-mipsel_24kc_musl/libgpiod-1.4.4/bindings/python'
/bin/sh ../../libtool <...> -I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9 <...>
In file included from /Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9/Python.h:63,
                 from gpiodmodule.c:8:
/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9/pyport.h:741:2: error: #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
 #error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
  ^~~~~
gmake[7]: *** [Makefile:525: gpiod_la-gpiodmodule.lo] Error 1

with the Makefile changes I get

PYTHON_CPPFLAGS="-I/Volumes/work/openwrt/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9" PYTHON_LIBS="-lpython3.9"

and later

gmake[7]: Entering directory '/Volumes/work/openwrt/build_dir/target-mipsel_24kc_musl/libgpiod-1.4.4/bindings/python'
/bin/sh ../../libtool <...> -I/Volumes/work/openwrt/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9 <...>
libtool: link: mipsel-openwrt-linux-musl-gcc-ar cr .libs/gpiod.a  gpiod_la-gpiodmodule.o
libtool: link: mipsel-openwrt-linux-musl-gcc-ranlib .libs/gpiod.a
libtool: link: ( cd ".libs" && rm -f "gpiod.la" && ln -s "../gpiod.la" "gpiod.la" )
gmake[7]: Leaving directory '/Volumes/work/openwrt/build_dir/target-mipsel_24kc_musl/libgpiod-1.4.4/bindings/python'

and a working module

some additional info:

k_ronny@pollux openwrt/ > /usr/bin/python3 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes               
-I/Library/Python/3.8/include -I/Library/Python/3.8/include

k_ronny@pollux openwrt/ > /usr/local/bin/python3 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
-I/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/include/python3.9

k_ronny@pollux openwrt/ > staging_dir/hostpkg/bin/python3 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
-I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9 -I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9

k_ronny@pollux openwrt/ > staging_dir/host/bin/python3 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
-I/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/usr/local/Cellar/python@3.9/3.9.1_8/Frameworks/Python.framework/Versions/3.9/include/python3.9
mhei commented 3 years ago

So there is definitely something wrong with your Python: python3-config is a simple shell script and should start with "#!/bin/sh" - it's not a Python script itself.

k-ronny commented 3 years ago

@mhei what can I say... on macOS

k_ronny@pollux openwrt/ > python3 
Python 3.9.1 (default, Feb  3 2021, 07:38:02) 
[Clang 12.0.0 (clang-1200.0.32.29)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

k_ronny@pollux openwrt/ > ls -l $(find staging_dir -name "pyth*config*")                                                                                                      
lrwxr-xr-x 1 k_ronny staff   16 Feb 11 16:07 staging_dir/hostpkg/bin/python3-config -> python3.9-config
-rwxr-xr-x 1 k_ronny staff 2,1K Feb 11 16:07 staging_dir/hostpkg/bin/python3.9-config
-rwxr-xr-x 1 k_ronny staff 2,1K Feb 11 16:07 staging_dir/hostpkg/lib/python3.9/config-3.9/python-config.py
-rwxr-xr-x 1 k_ronny staff 2,0K Jan 29 03:22 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
-rwxr-xr-x 1 k_ronny staff 2,0K Jan 29 03:22 staging_dir/target-mipsel_24kc_musl/usr/lib/python3.9/config-3.9/python-config.py

k_ronny@pollux openwrt/ > cat staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
#!/usr/bin/python3.9
# -*- python -*-

# Keep this script in sync with python-config.sh.in

import getopt
import os
import sys
import sysconfig
...

the shell scripts are gone??? bin/python3.9-config and lib/python3.9/config-3.9/python-config.py are the same

on Debian Buster it looks right

k_ronny@openwrt-build openwrt/ > python3       
Python 3.9.1+ (default, Feb  5 2021, 13:46:56) 
[GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.

k_ronny@openwrt-build openwrt/ > ls -l $(find staging_dir -name "pyth*config*")                                       
lrwxrwxrwx 1 k_ronny k_ronny   16  6. Jan 14:13 staging_dir/hostpkg/bin/python3-config -> python3.9-config
-rwxr-xr-x 1 k_ronny k_ronny 3,3K  6. Jan 14:13 staging_dir/hostpkg/bin/python3.9-config
-rwxr-xr-x 1 k_ronny k_ronny 2,1K  6. Jan 14:13 staging_dir/hostpkg/lib/python3.9/config-3.9/python-config.py
-rwxr-xr-x 1 k_ronny k_ronny 3,8K  6. Jan 14:19 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
-rwxr-xr-x 1 k_ronny k_ronny 2,0K  6. Jan 14:19 staging_dir/target-mipsel_24kc_musl/usr/lib/python3.9/config-3.9/python-config.py

k_ronny@openwrt-build openwrt/ > cat staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
#!/bin/sh

# Keep this script in sync with python-config.in

exit_with_usage ()
...

now I am cleaning up and rebuilding the whole thing, maybe my problem will go away

neheb commented 3 years ago

ping @ynezz @httpstorm Any idea here?

k-ronny commented 3 years ago

@mhei @neheb

in build_dir/hostpkg/Python-3.9.1/Makefile you can find this

...
python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
        @ # Substitution happens here, as the completely-expanded BINDIR
        @ # is not available in configure
        sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
        @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
        LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
        @ # On Darwin, always use the python version of the script, the shell
        @ # version doesn't use the compiler customizations that are provided
        @ # in python (_osx_support.py).
        @if test `uname -s` = Darwin; then \
                cp python-config.py python-config; \
        fi
...
httpstorm commented 3 years ago

@neheb For me it builds both with or without changing the Makefile for libgpiod. make package/feeds/packages/libgpiod/{clean,compile} -j 16

@k-ronny Would you be able to provide me with your .config, so I can test the same configuration? I have tried my existing setup for WRT3200ACM, as well as a blank build for TL-1043ND v4 with only libgpiod added. Both build successfully.

At this point I have the following suspects:

Here is some data from my environment:

./staging_dir/hostpkg/bin/python3-config --includes
-I/Volumes/aaa/openwrt/staging_dir/hostpkg/include/python3.9 -I/Volumes/aaa/openwrt/staging_dir/hostpkg/include/python3.9

./staging_dir/target-mips_24kc_musl/host/bin/python3.9-config --includes
-bash: ./staging_dir/target-mips_24kc_musl/host/bin/python3.9-config: /usr/bin/python3.9: bad interpreter: No such file or directory

head -1 ./staging_dir/hostpkg/bin/python3-config 
#!/Volumes/aaa/openwrt/staging_dir/hostpkg/bin/python3.9

head -1 staging_dir/target-mips_24kc_musl/host/bin/python3.9-config 
#!/usr/bin/python3.9

ls /usr/bin/python3.9
ls: /usr/bin/python3.9: No such file or directory

which -a python3
/usr/local/bin/python3
/usr/bin/python3

which -a python3.9
/usr/local/bin/python3.9

python3
Python 3.9.1 (default, Jan  8 2021, 17:17:43) 
[Clang 12.0.0 (clang-1200.0.32.28)] on darwin

Good luck!

k-ronny commented 3 years ago

@httpstorm this is my environment (I use only Homebrew, there is and never was a MacPorts installation)

k_ronny@pollux openwrt/ > echo $path
/usr/local/bin /usr/bin /bin

k_ronny@pollux openwrt/ > ./staging_dir/hostpkg/bin/python3-config --includes
-I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9 -I/Volumes/work/openwrt/staging_dir/hostpkg/include/python3.9

k_ronny@pollux openwrt/ > ./staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes
zsh: ./staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config: bad interpreter: /usr/bin/python3.9: no such file or directory

k_ronny@pollux openwrt/ > head -1 ./staging_dir/hostpkg/bin/python3-config
#!/Volumes/work/openwrt/staging_dir/hostpkg/bin/python3.9

k_ronny@pollux openwrt/ > head -1 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
#!/usr/bin/python3.9

k_ronny@pollux openwrt/ > ls /usr/bin/python3.9
ls: /usr/bin/python3.9: No such file or directory

k_ronny@pollux openwrt/ > which -a python3
/usr/local/bin/python3
/usr/bin/python3

k_ronny@pollux openwrt/ > which -a python3.9
/usr/local/bin/python3.9

k_ronny@pollux openwrt/ > python3 -VV
Python 3.9.1 (default, Feb  3 2021, 07:38:02) 
[Clang 12.0.0 (clang-1200.0.32.29)]

k_ronny@pollux openwrt/ > perl -v
This is perl 5, version 28, subversion 2 (v5.28.2) built for darwin-thread-multi-2level
(with 2 registered patches, see perl -V for more detail)

Copyright 1987-2019, Larry Wall
...

??? I can not get it to use the right include dir, it should use

<...>/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9

but it uses

<...>/staging_dir/hostpkg/include/python3.9

here is my .config

CONFIG_TARGET_ramips=y
CONFIG_TARGET_ramips_mt76x8=y
CONFIG_TARGET_ramips_mt76x8_DEVICE_mediatek_linkit-smart-7688=y
CONFIG_BUSYBOX_CUSTOM=y
CONFIG_BUSYBOX_CONFIG_STTY=y
# CONFIG_PACKAGE_dnsmasq is not set
# CONFIG_PACKAGE_firewall is not set
CONFIG_PACKAGE_gpiod-tools=y
CONFIG_PACKAGE_i2c-tools=y
# CONFIG_PACKAGE_ip6tables is not set
# CONFIG_PACKAGE_iptables is not set
CONFIG_PACKAGE_kmod-i2c-core=y
CONFIG_PACKAGE_kmod-i2c-mt7628=y
# CONFIG_PACKAGE_kmod-ip6tables is not set
# CONFIG_PACKAGE_kmod-ipt-conntrack is not set
# CONFIG_PACKAGE_kmod-ipt-core is not set
# CONFIG_PACKAGE_kmod-ipt-nat is not set
# CONFIG_PACKAGE_kmod-ipt-offload is not set
CONFIG_PACKAGE_kmod-ledtrig-netdev=y
# CONFIG_PACKAGE_kmod-lib-crc-ccitt is not set
CONFIG_PACKAGE_kmod-mtd-rw=m
# CONFIG_PACKAGE_kmod-nf-conntrack is not set
# CONFIG_PACKAGE_kmod-nf-conntrack6 is not set
# CONFIG_PACKAGE_kmod-nf-flow is not set
# CONFIG_PACKAGE_kmod-nf-ipt is not set
# CONFIG_PACKAGE_kmod-nf-ipt6 is not set
# CONFIG_PACKAGE_kmod-nf-nat is not set
# CONFIG_PACKAGE_kmod-nf-reject is not set
# CONFIG_PACKAGE_kmod-nf-reject6 is not set
# CONFIG_PACKAGE_kmod-ppp is not set
CONFIG_PACKAGE_kmod-usb-serial=y
CONFIG_PACKAGE_kmod-usb-serial-ftdi=y
CONFIG_PACKAGE_libbz2=y
CONFIG_PACKAGE_libffi=y
CONFIG_PACKAGE_libgpiod=y
CONFIG_PACKAGE_libi2c=y
# CONFIG_PACKAGE_libip4tc is not set
# CONFIG_PACKAGE_libip6tc is not set
CONFIG_PACKAGE_libiwinfo-lua=y
CONFIG_PACKAGE_liblua=y
CONFIG_PACKAGE_liblua5.3=y
CONFIG_PACKAGE_liboniondebug=y
CONFIG_PACKAGE_libonioni2c=y
CONFIG_PACKAGE_libonionoledexp=y
CONFIG_PACKAGE_librt=y
CONFIG_PACKAGE_libubox-lua=y
CONFIG_PACKAGE_libubus-lua=y
CONFIG_PACKAGE_libuci-lua=y
CONFIG_PACKAGE_libuuid=y
# CONFIG_PACKAGE_libxtables is not set
CONFIG_PACKAGE_lua=y
CONFIG_PACKAGE_lua-periphery=y
CONFIG_PACKAGE_lua-periphery5.3=y
CONFIG_PACKAGE_lua5.3=y
CONFIG_PACKAGE_luasocket=y
CONFIG_PACKAGE_luasocket5.3=y
CONFIG_PACKAGE_netcat=y
# CONFIG_PACKAGE_odhcp6c is not set
# CONFIG_PACKAGE_odhcpd-ipv6only is not set
CONFIG_PACKAGE_oled-exp=y
# CONFIG_PACKAGE_ppp is not set
CONFIG_PACKAGE_procd-seccomp=y
CONFIG_PACKAGE_python3-base=y
CONFIG_PACKAGE_python3-ctypes=y
CONFIG_PACKAGE_python3-gpiod=y
CONFIG_PACKAGE_python3-light=y
CONFIG_PACKAGE_python3-oled-exp=y
CONFIG_PACKAGE_python3-omega-expansion=y
CONFIG_PACKAGE_python3-onion-i2c=y
CONFIG_PACKAGE_python3-periphery=y
CONFIG_PACKAGE_python3-smbus=y
CONFIG_PACKAGE_rpcd=y
CONFIG_PACKAGE_rpcd-mod-file=y
CONFIG_PACKAGE_rpcd-mod-iwinfo=y
CONFIG_PACKAGE_rpcd-mod-rpcsys=y
CONFIG_PACKAGE_umdns=y
CONFIG_PACKAGE_zlib=y
httpstorm commented 3 years ago

@k-ronny Can you please provide the entire .config? Perhaps post it on pastern.com One thing we can test is changing ./staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config, to use perl from /Volumes/work/openwrt/staging_dir/hostpkg/bin/python3.9. Do you know which code is used to generate this script?

My brand new MacBook Pro 16 is totally borked at the moment, and I need to show it to the Apple support in about two hours before I can restart it. It's been making kernel panics whenever I restart it. I might be able to do some testing after that, and once I have the entire .config.

k-ronny commented 3 years ago

@httpstorm I think there is no way to get the right include dir with this python version of the config script. It will always print out options for the build host python and not for the target. And I do not know how perl is involved here I see only some sed magic in <...>/build_dir/hostpkg/Python-3.9.1/Makefile.pre.in line 1578ff.

# bpo-21536: Misc/python-config.sh is generated in the build directory
# from $(srcdir)Misc/python-config.sh.in.
python-config: $(srcdir)/Misc/python-config.in Misc/python-config.sh
        @ # Substitution happens here, as the completely-expanded BINDIR
        @ # is not available in configure
        sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
        @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
        LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config

If I enable the installation of the shell script version all is good.

e.g. <...>/feeds/packages/lang/python/python3/patches/666-macos-config.patch

--- a/Makefile.pre.in   2021-02-12 00:16:20.000000000 +0100
+++ b/Makefile.pre.in   2021-02-12 17:44:08.000000000 +0100
@@ -1583,12 +1583,6 @@
    sed -e "s,@EXENAME@,$(BINDIR)/python$(LDVERSION)$(EXE)," < $(srcdir)/Misc/python-config.in >python-config.py
    @ # Replace makefile compat. variable references with shell script compat. ones; $(VAR) -> ${VAR}
    LC_ALL=C sed -e 's,\$$(\([A-Za-z0-9_]*\)),\$$\{\1\},g' < Misc/python-config.sh >python-config
-   @ # On Darwin, always use the python version of the script, the shell
-   @ # version doesn't use the compiler customizations that are provided
-   @ # in python (_osx_support.py).
-   @if test `uname -s` = Darwin; then \
-       cp python-config.py python-config; \
-   fi

 # Install the include files

But this means it will not use macOS specific options. Is this a problem?

httpstorm commented 3 years ago

Hello @k-ronny Please provide me with the full .config file, so I can try to build the same configuration! You might also try building from my fork:

git clone https://github.com/httpstorm/openwrt.git
git checkout gvalkov

Before building, add the patches from this PR to feeds/packages/libs/perl/patches https://github.com/openwrt/packages/pull/14676/files

I don't think perl is involved in the python build. I mentioned it, because it itself fails to build on macOS. Accidentally the perl build fails because it fails to find the current directory (cwd).

Here is my list of installed brew and MacPorts packages: https://pastebin.com/MvWrMpAP

Good luck!

k-ronny commented 3 years ago

@httpstorm o.k. you can download my .config from here: config.txt but I am afraid there are some packages I have added that are not available to you.

My brew installed packages are

and it's dependencies.

I have just started a build from your tree.

httpstorm commented 3 years ago

Hello @k-ronny ! I reproduced your issue using the .config you provided. Then I tried to compile the bare minimum:

The build succeeded. Then I added:

The build failed reproducing your issue. I will look into this soon later today or tomorrow. Good luck!

k-ronny commented 3 years ago

@httpstorm oh I already know the reason for this, see https://github.com/openwrt/packages/issues/14652#issuecomment-777591988. <...>/build_dir/target-mipsel_24kc_musl/libgpiod-1.4.4/bindings/python/gpiodmodule.c includes <...>/staging_dir/hostpkg/include/python3.9/Python.h which includes <...>/staging_dir/hostpkg/include/python3.9/pyport.h where you can find this

...
#ifndef LONG_BIT
#define LONG_BIT (8 * SIZEOF_LONG)
#endif

#if LONG_BIT != 8 * SIZEOF_LONG
/* 04-Oct-2000 LONG_BIT is apparently (mis)defined as 64 on some recent
 * 32-bit platforms using gcc.  We try to catch that here at compile-time
 * rather than waiting for integer multiplication to trigger bogus
 * overflows.
 */
#error "LONG_BIT definition appears wrong for platform (bad gcc/glibc config?)."
#endif
...

and pyport.h includes <...>/staging_dir/hostpkg/include/python3.9/pyconfig.h where you can find this

/* The size of `long', as computed by sizeof. */
#define SIZEOF_LONG 8

I am compiling on a platform where LONG is 8 (macOS) for a platform where LONG is 4 (mipsel)

so the compiler should use <...>/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9/pyport.h and than <...>/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9/pyconfig.h where LONG is defined as

/* The size of `long', as computed by sizeof. */
#define SIZEOF_LONG 4
jefferyto commented 3 years ago

Thanks @k-ronny for debugging this - I think if we install the shell script version of python-config to $(STAGING_DIR)/host/bin on macOS that would solve your build error, is this correct?

I think it makes sense to install the shell script version for target Python, since it is used to help cross-compile other packages and so no Mac-specific customizations should be used. The shell script version should also be packaged in python3-dev, to be consistent with Linux builds. For host Python (installed to $(STAGING_DIR_HOSTPKG)/bin) I think it should stay as the Python version.

@commodo any thoughts?

commodo commented 3 years ago

Thanks @k-ronny for debugging this - I think if we install the shell script version of python-config to $(STAGING_DIR)/host/bin on macOS that would solve your build error, is this correct?

I think it makes sense to install the shell script version for target Python, since it is used to help cross-compile other packages and so no Mac-specific customizations should be used. The shell script version should also be packaged in python3-dev, to be consistent with Linux builds. For host Python (installed to $(STAGING_DIR_HOSTPKG)/bin) I think it should stay as the Python version.

@commodo any thoughts?

works for me; i have nothing to add

httpstorm commented 3 years ago

Hello @k-ronny Your fix is very elegant, and we should implement it. I can confirm, that it works correctly on both macOS and Ubuntu Linux. -I$(PYTHON3_INC_DIR) is already in use by a few other packages. There is also a version of this macro for the host: HOST_PYTHON3_INC_DIR. The make variables are defined respectively in python3-package.mk, and python3-host.mk. Can you make a pull-request with your patch or would you like me to do it?

Your observations are right on the point! The reason why the following check fails #if LONG_BIT != 8 * SIZEOF_LONG, is because LONG_BIT = 32 is defined in limits.h for the target architecture of your router, while SIZEOF_LONG = 8 comes from the version of python compiled for the host. The build for the target is using headers from the host package which is incorrect. ./staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config --includes should return -I/home/g/test/ccc/openwrt/staging_dir/target-mipsel_24kc_musl/usr/include/python3.9 but there are multiple issues in the script:

  1. The shebang #!/usr/bin/python3.9 points to a read only file system on macOS, so python cannot be installed there. The only alternative is generating the script with a valid python3.9 path.
  2. Even if the shebang is corrected, the script lacks prefix and prefix_real, which are defined when building on Linux. As a result, the script returns different include paths specific to the instance of python in the shebang.
  3. On Linux, python3.9-config is using a shell script, while on macOS it is a python script.

We should fix this script to include a proper shebang, and a valid prefix. Using a shell script is preferred to be consistent on both platforms.

httpstorm commented 3 years ago

Thanks @k-ronny for debugging this - I think if we install the shell script version of python-config to $(STAGING_DIR)/host/bin on macOS that would solve your build error, is this correct?

I think it makes sense to install the shell script version for target Python, since it is used to help cross-compile other packages and so no Mac-specific customizations should be used. The shell script version should also be packaged in python3-dev, to be consistent with Linux builds. For host Python (installed to $(STAGING_DIR_HOSTPKG)/bin) I think it should stay as the Python version.

@jefferyto @commodo Sounds good to me. Can you please implement this? It would otherwise take me some time to get familiar with the python build.

mhei commented 3 years ago

@httpstorm: I'm confused with your earlier comment

-I$(PYTHON3_INC_DIR) is already in use by a few other packages

and where you propose to implement this as a fix? At the moment, I'd NAK this since in my eyes the problem here is that python development environment is mis-configured on macOS. So I'd agree with @jefferyto proposal too, i.e. to fixup the python-config script. But maybe I got you wrong, then sorry...

httpstorm commented 3 years ago

@mhei We have two issues:

  1. python-config is misconfigured on macOS.
  2. libgpiod not building on macOS because of 1.

The patch suggested by @k-ronny in his initial post https://github.com/openwrt/packages/issues/14652#issue-801421128 is a valid solution to point 2. -I$(PYTHON3_INC_DIR) is already in use by other projects. Of course we must address point 1 with priority. I will look into the python3.9-config script now.

grep -r "PYTHON3_INC_DIR" .
./feeds/packages/net/uwsgi/Makefile:        CFLAGS="$(TARGET_CPPFLAGS) -I$(PYTHON3_INC_DIR) $(TARGET_CFLAGS)" \
./feeds/packages/lang/python/python3-libsemanage/Makefile:  PYINC="-I $(PYTHON3_INC_DIR)" \
./feeds/packages/lang/python/python3-package.mk:PYTHON3_INC_DIR:=$(PYTHON3_DIR)/include/python$(PYTHON3_VERSION)
./feeds/packages/lang/python/python3-package.mk:    CPPFLAGS="$(TARGET_CPPFLAGS) -I$(PYTHON3_INC_DIR)" \
./feeds/packages/lang/python/python3-libselinux/Makefile:   PYINC="-I $(PYTHON3_INC_DIR)" \
./feeds/packages/lang/python/python3-host.mk:HOST_PYTHON3_INC_DIR:=$(HOST_PYTHON3_DIR)/include/python$(PYTHON3_VERSION)
./feeds/packages/lang/python/python3-host.mk:   CPPFLAGS="$(HOST_CPPFLAGS) -I$(HOST_PYTHON3_INC_DIR)" \
./package/feeds/packages/python3-libsemanage/Makefile:  PYINC="-I $(PYTHON3_INC_DIR)" \
./package/feeds/packages/python3-libselinux/Makefile:   PYINC="-I $(PYTHON3_INC_DIR)" \
./package/feeds/packages/uwsgi/Makefile:        CFLAGS="$(TARGET_CPPFLAGS) -I$(PYTHON3_INC_DIR) $(TARGET_CFLAGS)" \
k-ronny commented 3 years ago

Hi, based on this discussion I now use this patch <...>feeds/packages/lang/python/python3/patches/666-macos-config.patch

--- a/Makefile.pre.in   2021-02-15 15:58:26.000000000 +0100
+++ b/Makefile.pre.in   2021-02-15 16:16:10.000000000 +0100
@@ -1586,7 +1586,7 @@
    @ # On Darwin, always use the python version of the script, the shell
    @ # version doesn't use the compiler customizations that are provided
    @ # in python (_osx_support.py).
-   @if test `uname -s` = Darwin; then \
+   @if test `uname -s` = Darwin -a "$(STAGING_DIR_HOSTPKG)" = "$(STAGING_PREFIX)"; then \
        cp python-config.py python-config; \
    fi

this patch results in

k_ronny@pollux openwrt/ > head -1 staging_dir/target-mipsel_24kc_musl/host/bin/python3.9-config
#!/bin/sh

k_ronny@pollux openwrt/ > head -1 staging_dir/hostpkg/bin/python3.9-config
#!/Volumes/work/openwrt/staging_dir/hostpkg/bin/python3.9

and solves my problem without patching libgpiod.

httpstorm commented 3 years ago

@k-ronny I have tested your last patch https://github.com/openwrt/packages/issues/14652#issuecomment-779484949. It compiles correctly on both Linux and macOS. When building python3/host on macOS, the python version of the config script is used, which enables compiler optimisations, since the python script returns configuration for the running python interpreter. This configuration however does not apply to the target and results in build errors. Your last patch resolves the errors by using the shell script for the target build, which returns the correct configuration.

Please name your patch according to these rules, and create a pull-request: https://openwrt.org/docs/guide-developer/build-system/use-patches-with-buildsystem#naming_patches

jefferyto commented 3 years ago

Thanks again @k-ronny for your work on this. Your patch is functionally the same as #14767 but I think #14767 is slightly more readable/intentional.

k-ronny commented 3 years ago

Hi, of course, many thanks to all. I am happy that this issue has been solved.

httpstorm commented 3 years ago

Thank you, @k-ronny for the hard work and research, and @jefferyto for the fix! Good luck!

mhei commented 3 years ago

From me, too: thank you all for your hard work and time on this issue.