Compile Python 2.7.2 with WASM

[yzhang71]

We are currently attempting to compile Python version 2.7.2 with our Lind-WASM. We selected version 2.7.2 because it is the version used by Lind-NaCl, allowing for a fair comparison. I will also use this issue to track the compilation problems and document their solutions.

[yzhang71]

In file included from ../Modules/python.c:3: In file included from ../Include/Python.h:58: ../Include/pyport.h:243:13: error: "This platform's pyconfig.h needs to define PY_FORMAT_LONG_LONG" error "This platform's pyconfig.h needs to define PY_FORMAT_LONG_LONG" ^ 1 error generated. make: *** [Makefile:555: Modules/python.o] Error 1

[rennergade]

Where did you pull the source code from?

[yzhang71]

The error persists because we are performing a cross-compilation, and PY_FORMAT_LONG_LONG is not correctly defined for the platform. To resolve this issue in a robust way, we need to define PY_FORMAT_LONG_LONG directly in pyconfig.h.

Solution:

Add the following line to /Python-2.7.2/build/pyconfig.h:

#define PY_FORMAT_LONG_LONG "ll"

[yzhang71]

Where did you pull the source code from?

I got the source code from here: https://www.python.org/downloads/release/python-272/

[rennergade]

A lot of clues for things such as the one above should be in here: https://github.com/Lind-Project/lind_project/tree/main/tests/applications/python

Python was the hardest app to build because its build process uses itself to package everything, but because we couldn't switch between native and Lind in the build process we had to compile a bunch of things by scratch. It would be cool to figure it out seemlessly this time but that may not be possible.

[Yaxuan-w]

This issue would be a good reference from my view

[yzhang71]

Parser/pgen ../Grammar/Grammar ../Include/graminit.h ../Python/graminit.c Parser/pgen: 12: Syntax error: "(" unexpected make: *** [Makefile:562: Parser/pgen.stamp] Error 2

[yzhang71]

The error occurred because the pgen we generated was a WASM version. However, the default script attempts to run pgen directly, without using a WASM runtime.

Solution

Inspired by the Lind-NaCl compilation process, I followed these steps to resolve the issue:

1. Compiled a native version of Python using Clang.
2. Used the native pgen to generate the necessary files:
   • ../Include/graminit.h
   • ../Python/graminit.c

[yzhang71]

../Modules/posixmodule.c:5873:21: error: use of undeclared identifier 'MAX_GROUPS' gid_t grouplist[MAX_GROUPS]; ^ ../Modules/posixmodule.c:5880:15: error: use of undeclared identifier 'MAX_GROUPS' if (len > MAX_GROUPS) {

[yzhang71]

The error occurs because MAX_GROUPS is not defined for our platform during compilation. To fix this issue, we can manually define MAX_GROUPS in the pyconfig.h file.

Solution

Add the following line to the pyconfig.h file in your build directory:

#define MAX_GROUPS 64

Explanation

MAX_GROUPS specifies the maximum number of groups a user can belong to.

[yzhang71]

../Modules/posixmodule.c:1800:11: error: call to undeclared function 'chflags'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration] res = chflags(path, flags); ^

[yzhang71]

The error occurs because the function chflags is not declared or supported on our platform.

Solution: Disable chflags

Modify the build to exclude chflags if it is not supported on your platform. In the pyconfig.h file located in your build directory, ensure that HAVE_CHFLAGS is undefined:

   #undef HAVE_CHFLAGS

[yzhang71]

    Modules/python.o \
    libpython2.7.a    -lm  

wasm-ld: error: unable to find library -lm clang-16: error: linker command failed with exit code 1 (use -v to see invocation) make: *** [Makefile:416: python] Error 1

[yzhang71]

This error happens because the linker (wasm-ld) cannot locate the libm library, which provides mathematical functions.

Solution

To resolve the issue, create a static libm.a library in the sysroot location for WASM: Run the following command to generate libm.a:

   /clang+llvm-16.0.4-x86_64-linux-gnu-ubuntu-22.04/bin/llvm-ar crs "/lind-wasm/glibc/sysroot/lib/wasm32-wasi/libm.a"

Explanation

• The -lm flag instructs the linker to link against the math library (libm).
• For WASM, a compatible libm.a must exist in the sysroot's library path (/lib/wasm32-wasi/).
• Running llvm-ar crs creates an empty libm.a, which satisfies the linker's requirement for the library, even if the actual mathematical functions are not needed.