search

singlestore-labs / python-wasi

Utilities for building CPython for the WASI platform
Apache License 2.0
97 stars 14 forks source link

./Include/pythread.h:130:13: fatal error: 'pthread.h' file not found #1

Closed AliceNZ closed 2 years ago

AliceNZ commented 2 years ago

Hi, good day there. I followed the readme with the following commands but I encountered errors during the build: $docker build -f docker/Dockerfile -t wasi-build:latest docker $docker run -it --rm -v $(pwd):$(pwd) -w $(pwd) wasi-build:latest bash $./run.sh ... 1st error: building '_ctypes' extension gcc -pthread -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 ... ~/python-wasi-main/cpython/Modules/_ctypes/_ctypes.c:117:10: fatal error: ffi.h: No such file or directory 117 | #include | ^~~ compilation terminated.

Solved by $apt install libffi-dev

2nd error: In file included from ./Modules/_ctypes/_ctypes_test.c:1: In file included from ./Include/Python.h:88: ./Include/pythread.h:130:13: fatal error: 'pthread.h' file not found

include

Don't know how to solve it, do you have any suggestions?

kesmit13 commented 2 years ago

Are you running the ./run.sh inside the Docker container? The messages here look a bit off because it should be using clang rather than gcc. pthread.h should be picked up from the wasix subdirectory, which should have been cloned when you executed the run.sh script. The ctypes library still won't work though because WASM doesn't support dynamic linking yet.

AliceNZ commented 2 years ago

Yes, I ran the ./run.sh inside the Docker container and it used clang to build. 

$ docker run -it --rm -v $(pwd):$(pwd) -w $(pwd) wasi-build:latest bash
root@#  ./run.sh
......
clang --target=wasm32-wasi -c -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -I/opt/include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -isystem /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -I/opt/include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -isystem /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include  -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include    -DPy_BUILD_CORE -DPYTHONPATH='""' \
    -DPREFIX='"/"' \
    -DEXEC_PREFIX='"/"' \
    -DVERSION='"3.11"' \
    -DVPATH='""' \
    -DPLATLIBDIR='"lib"' \
    -o Modules/getpath.o ./Modules/getpath.c
./Modules/expat/xmlparse.c:3909:9: warning: code will never be executed [-Wunreachable-code]
        parser->m_characterDataHandler(parser->m_handlerArg, parser->m_dataBuf,
        ^~~~~~
./Modules/expat/xmlparse.c:3908:16: note: silence by adding parentheses to mark code as explicitly dead
      else if (0 && parser->m_characterDataHandler)
               ^
               /* DISABLES CODE */ ( )
LC_ALL=C sed -e 's,\$(\([A-Za-z0-9_]*\)),\$\{\1\},g' < Misc/python-config.sh >python-config
./Modules/_ctypes/_ctypes_test.c:1:
In file included from ./Include/Python.h:88:
./Include/pythread.h:130:13: fatal error: 'pthread.h' file not found
#   include <pthread.h>
            ^~~~~~~~~~~
1 error generated.
......
5 warnings generated.
Error: failed to run main module `cpython/python.wasm`

Caused by:
    0: failed to open: cpython/python.wasm
    1: No such file or directory (os error 2)
root@#
kesmit13 commented 2 years ago

Is there a wasix subdirectory where you ran ./run.sh? That would contain the include/pthread.h file.

AliceNZ commented 2 years ago

yes, there is wasix folder containing the include/pthread.h file. My OS is Ubuntu 20.04 LTS 64-bit.

kesmit13 commented 2 years ago

Try this for now. Go into the cpython directory and do git checkout 3.10. That will take you off of the current Python code back to a stable release.

AliceNZ commented 2 years ago

Tried but failed with different errors: python-wasi-main# cd cpython/ cpython# git checkout 3.10 cpython# git pull cpython# cd .. python-wasi-main#./run.sh clang --target=wasm32-wasi -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -I/opt/include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -isystem /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -I/opt/include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -isystem /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include -I. -I/opt/wasi-sdk-14.0/bin/../share/wasi-sysroot/include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython -c /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Modules/syslogmodule.c -o build/temp.wasi-wasm32-3.10/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Modules/syslogmodule.o In file included from /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Modules/mmapmodule.c:55: /opt/wasi-sdk-14.0/bin/../share/wasi-sysroot/include/sys/mman.h:2:2: error: "WASI lacks a true mmap; to enable minimal mmap emulation, compile with -D_WASI_EMULATED_MMAN and link with -lwasi-emulated-mman"

error "WASI lacks a true mmap; to enable minimal mmap emulation, \

^ /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Modules/grpmodule.c:7:10: fatal error: 'grp.h' file not found /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Modules/mmapmodule.c:160:9: #include error:

fatal error: too many errors emitted, stopping now [-ferror-limit=] 20 errors generated. building '_testcapi' extension clang --target=wasm32-wasi -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -I/opt/include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -isystem /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -I/opt/include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -isystem /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/wasix/include -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I./Include -I. -I/opt/wasi-sdk-14.0/bin/../share/wasi-sysroot/include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Include -I/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython -c /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Modules/_testcapimodule.c -o build/temp.wasi-wasm32-3.10/home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Modules/_testcapimodule.o /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Modules/syslogmodule.c:55:10: fatal error: 'syslog.h' file not found

include

...... Failed to build these modules: _ctypes _ctypes_test _multiprocessing
_opcode _testbuffer _testcapi
_testimportmultiple _testmultiphase _xxsubinterpreters _xxtestfuzz audioop grp
mmap resource syslog
termios xxlimited xxlimited_35

Could not build the ssl module! Python requires a OpenSSL 1.1.1 or newer

running build_scripts copying and adjusting /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Tools/scripts/pydoc3 -> build/scripts-3.10 copying and adjusting /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Tools/scripts/idle3 -> build/scripts-3.10 copying and adjusting /home/aliceou/Eratos/WebAssemblyStudy/WasmerPython/python-wasi/python-wasi-main/cpython/Tools/scripts/2to3 -> build/scripts-3.10 changing mode of build/scripts-3.10/pydoc3 from 644 to 755 changing mode of build/scripts-3.10/idle3 from 644 to 755 changing mode of build/scripts-3.10/2to3 from 644 to 755 renaming build/scripts-3.10/pydoc3 to build/scripts-3.10/pydoc3.10 renaming build/scripts-3.10/idle3 to build/scripts-3.10/idle3.10 renaming build/scripts-3.10/2to3 to build/scripts-3.10/2to3-3.10 #

kesmit13 commented 2 years ago

Strange. I'm seeing these errors now too with a completely fresh build of the Docker image. I'm looking into it.

kesmit13 commented 2 years ago

It looks like the main branch has had some changes (https://bugs.python.org/issue45847) that are preventing the configure.ac patch to be applied completely. You should still be able to check out previous version branches though. I was able to build Python 3.10 without issue. Keep in mind that not everything is going to build. All of the modules you have listed in your last message have dependencies that don't exist in WASI yet, so they won't compile.

kesmit13 commented 2 years ago

I made a smaller patch to configure.ac that works now (004e81553cf487bb4d8e5454f82fb3f7306a2ec8).