build fails (pyston, Python.h not found)

[belm0]

Describe the bug Build targeting pyston (API-compatible fork of CPython) fails.

In some compiler invocations, the python include directory is correctly present, but in others it isn't.

For example, this is OK:

gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -g -O2 -fdebug-prefix-map=/src/build/pyston=. -fstack-protector-strong -Wformat -Werror=format-security -DENABLE_AOT -g -O2 -fdebug-prefix-map=/src/build/pyston=. -fstack-protector-strong -Wformat -Werror=format-security -fPIC -I. -Ivendor/Heap-Layers -Ivendor/Heap-Layers/utility -I/usr/include/python3.8-pyston2.3 -c src/source/get_line_atomic.cpp -o build/temp.linux-x86_64-3.8-pyston2.3/src/source/get_line_atomic.o -std=c++14

this is not (-I/usr/include/python3.8-pyston2.3 is missing):

creating build/temp.linux-x86_64-3.8-pyston2.3/scalene
make OUTDIR=build/temp.linux-x86_64-3.8-pyston2.3/scalene ARCH=
make: python3-config: Command not found
g++ -std=c++17 -Wall -g -O3 -DNDEBUG -D_REENTRANT=1 -DHL_USE_XXREALLOC=1 -pipe -fno-builtin-malloc -fvisibility=hidden -fPIC -shared -Bsymbolic -Isrc -Isrc/include -Ivendor/Heap-Layers -Ivendor/Heap-Layers/wrappers -Ivendor/Heap-Layers/utility -Ivendor/printf  -I/usr/include/nptl  src/source/libscalene.cpp vendor/Heap-Layers/wrappers/gnuwrapper.cpp vendor/printf/printf.cpp -o build/temp.linux-x86_64-3.8-pyston2.3/scalene/libscalene.so -ldl -lpthread
src/source/libscalene.cpp:94:10: fatal error: Python.h: No such file or directory

To Reproduce

docker run --rm -it pyston/pyston bash

$ pip install wheel
$ pip install scalene
...
    g++ -std=c++17 -Wall -g -O3 -DNDEBUG -D_REENTRANT=1 -DHL_USE_XXREALLOC=1 -pipe -fno-builtin-malloc -fvisibility=hidden -fPIC -shared -Bsymbolic -Isrc -Isrc/include -Ivendor/Heap-Layers -Ivendor/Heap-Layers/wrappers -Ivendor/Heap-Layers/utility -Ivendor/printf  -I/usr/include/nptl  src/source/libscalene.cpp vendor/Heap-Layers/wrappers/gnuwrapper.cpp vendor/printf/printf.cpp -o build/temp.linux-x86_64-3.8-pyston2.3/scalene/libscalene.so -ldl -lpthread
    src/source/libscalene.cpp:94:10: fatal error: Python.h: No such file or directory

Expected behavior successful package build

[belm0]

make: python3-config: Command not found

I see, the scalene build is depending on python3-config, pyston should provide it somehow

[belm0]

It might be better for the scalene build to get flags from the interpreter than invoke the build. Besides eliminating the dependency on python3-config, it would support environments with multiple python3 versions (ensuring that the correct headers were used).

e.g. sysconfig.get_path('include')

[AvdN]

I think I ran into the same problem trying to install scalene in a virtualenv. That was on Ubuntu 22.04 which has a system Python 3.10, and no other python stuff installed with apt (so no header files for Python 3.10). I compiled Python 3.12.1 succesfully, installed it under /opt/python/3.12.1 and created a virtualenv /opt/util/scalene from that.

So far so good, but doing pip install scalene resulted in Python.h not found. Some modules were correctly compiled with -I/opt/python/3.12.1/include/python3.12 but the failing one had -I/usr/include/python3.10 a directory that doesn't even exist on the system.

So there is definately something wrong with how this is compiled, I have no such problems with the C extension for my YAML parser (ruamel.yaml.clib, which as worked for Python2.5+).

The way around this that worked for me, is activating the virtual env (source /opt/util/scalene/bin/activate) and then running pip install scalene, but that should not have been necessary

part of the erroneous compile output

      g++ -std=c++14 -Wall -g -O3 -DNDEBUG -D_REENTRANT=1 -DHL_USE_XXREALLOC=1 -pipe -fno-builtin-malloc -fvisibility=hidden -fPIC -shared -Bsymbolic -Isrc -Isrc/include -Ivendor 
/Heap-Layers -Ivendor/Heap-Layers/wrappers -Ivendor/Heap-Layers/utility -Ivendor/printf -I/usr/include/python3.10 -I/usr/include/nptl  src/source/libscalene.cpp vendor/Heap-Layer 
s/wrappers/gnuwrapper.cpp vendor/printf/printf.cpp -o build/temp.linux-x86_64-cpython-312/scalene/libscalene.so -ldl -lpthread
      src/source/libscalene.cpp:93:10: fatal error: Python.h: No such file or directory
         93 | #include <Python.h>
            |          ^~~~~~~~~~

[chuckwondo]

Try installing these system packages in your docker image: python3-dev musl-dev linux-headers