Invalid include flags when compiling Cython extension

[touilleMan]

tl;dr: Last release 20221220 fails to compile a simple cython project:

setup.py:

from setuptools import setup
from Cython.Build import cythonize

ext_modules = cythonize("my.pyx")
setup(
    ext_modules=ext_modules,
)

(my.pyx can be an empty file)

With 20221220/cpython-3.10.9+20221220-x86_64-unknown-linux-gnu-pgo+lto-full.tar.zst:

/tmp/godot-python-test-90vls0a3/addons/pythonscript/linux-x86_64/bin/python3 setup.py build_ext --build-lib /tmp/godot-python-test-90vls0a3
Compiling my.pyx because it changed.
[1/1] Cythonizing my.pyx
running build_ext
building 'my' extension
creating build
creating build/temp.linux-x86_64-cpython-310
clang -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -fPIC -I/install/include/python3.10 -c my.c -o build/temp.linux-x86_64-cpython-310/my.o
my.c:19:10: fatal error: 'Python.h' file not found
#include "Python.h"
         ^~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1

vs with 20221106/cpython-3.10.8+20221106-x86_64-unknown-linux-gnu-pgo+lto-full.tar.zst

/tmp/godot-python-test-n_svlsv_/addons/pythonscript/linux-x86_64/bin/python3 setup.py build_ext --build-lib /tmp/godot-python-test-n_svlsv_
Compiling my.pyx because it changed.
[1/1] Cythonizing my.pyx
running build_ext
building 'my' extension
creating build
creating build/temp.linux-x86_64-3.10
clang -pthread -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -fPIC -I/tmp/godot-python-test-n_svlsv_/addons/pythonscript/linux-x86_64/include/python3.10 -c my.c -o build/temp.linux-x86_64-3.10/my.o
clang -pthread -shared -L/tools/deps/lib -Wl,--exclude-libs,ALL -L/tools/deps/libedit/lib build/temp.linux-x86_64-3.10/my.o -L/install/lib -o /tmp/godot-python-test-n_svlsv_/my.cpython-310-x86_64-linux-gnu.so
rename my.cpython-310-x86_64-linux-gnu.so -> my.so

The issue seems to be the includes flags end up with an empty prefix path (hence e.g. -I/install/include/python3.10 instead of -I/my/path/to/python-build-standalone/install/include/python3.10)

I'm not sure what the reason given python3-config output the correct include flags:

$ /tmp/godot-python-test-90vls0a3/addons/pythonscript/linux-x86_64/bin/python3-config
-I/home/emmanuel/projects/godot-python/build/common_tests_install_distrib/addons/pythonscript/linux-x86_64/include/python3.10 -I/home/emmanuel/projects/godot-python/build/common_tests_install_distrib/addons/pythonscript/linux-x86_64/include/python3.10

[jvolkman]

Take a look at https://gregoryszorc.com/docs/python-build-standalone/main/quirks.html#references-to-build-time-paths.

[adrian-kong]

Hey @touilleMan , were you able to resolve this issue?

I'm running into the exact same situation but on 3.11 (works fine on 3.9 and 3.10) Error occured using pip install which I managed to fix by adjusting CONFINCLUDEPY pointing to /install to absolute path of that folder in _sysconfigdata_*.py

Just wondering how you manage to get it working or if there was a way to not modify CONFINCLUDEPY.

[touilleMan]

Hi @adrian-kong ,

I've found a workaround for this issue: find by hand the actual include dir of the python distribution I want to compile against, then provide it as an additional include.

https://github.com/touilleMan/godot-python/blob/f2c09ae937dd53634c98fc9309d10209138d9577/tests/3-pythonscript-cython-only/setup.py#L7-L22

This is not perfect given the incorrect include flags a still present and can take priority if they point to a valid location :'(

The final commands setuptools runs to compile my cython extension (/tmp/godot-python-test-qzmy_izf is the working folder, addons/pythonscript/linux-x86_64/ contains the Python distribution provided by python-build-standalone):

clang -pthread -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I/tools/deps/include -I/tools/deps/include/ncursesw -I/tools/deps/libedit/include -fPIC -Iaddons/pythonscript/linux-x86_64/include/python3.11 -Igdextension_api -I/install/include/python3.11 -c my.c -o build/temp.linux-x86_64-cpython-311/my.o
clang -pthread -shared -L/tools/deps/lib -Wl,--exclude-libs,ALL -L/tools/deps/libedit/lib build/temp.linux-x86_64-cpython-311/my.o -L/install/lib -o /tmp/godot-python-test-qzmy_izf/my.cpython-311-x86_64-linux-gnu.so