search

singlestore-labs / python-wasi

Utilities for building CPython for the WASI platform
Apache License 2.0
96 stars 14 forks source link

Docker build error: Cannot find `libc6-dev-i386` #2

Open technosophos opened 2 years ago

technosophos commented 2 years ago

With a fresh clone of the repo, running the Docker build gives me the following error:

 => CACHED [ 1/16] FROM docker.io/library/debian:bullseye@sha256:10b622c6cf6daa0a295be74c0e412ed20e10f91ae4c6f3ce6ff0c9c04f77cbf6    0.0s
 => ERROR [ 2/16] RUN apt-get update && apt-get install -y     wget     curl     lsb-release     software-properties-common     gnu  1.8s
------
 > [ 2/16] RUN apt-get update && apt-get install -y     wget     curl     lsb-release     software-properties-common     gnupg     vim     build-essential     libc6-dev     libc6-dev-i386     libtinfo5     m4     zip     git     cmake     ninja-build:
#5 0.165 Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB]
#5 0.186 Get:2 http://deb.debian.org/debian bullseye-updates InRelease [39.4 kB]
#5 0.216 Get:3 http://security.debian.org/debian-security bullseye-security InRelease [44.1 kB]
#5 0.256 Get:4 http://deb.debian.org/debian bullseye/main arm64 Packages [8070 kB]
#5 0.464 Get:5 http://security.debian.org/debian-security bullseye-security/main arm64 Packages [120 kB]
#5 0.526 Get:6 http://deb.debian.org/debian bullseye-updates/main arm64 Packages [2596 B]
#5 1.170 Fetched 8392 kB in 1s (8163 kB/s)
#5 1.170 Reading package lists...
#5 1.433 Reading package lists...
#5 1.682 Building dependency tree...
#5 1.744 Reading state information...
#5 1.795 E: Unable to locate package libc6-dev-i386
------
executor failed running [/bin/sh -c apt-get update && apt-get install -y     wget     curl     lsb-release     software-properties-common     gnupg     vim     build-essential     libc6-dev     libc6-dev-i386     libtinfo5     m4     zip     git     cmake     ninja-build]: exit code: 100
technosophos commented 2 years ago

Ah.... it took the second error to figure out what was really happening:

docker build -f docker/Dockerfile -t wasi-build:latest docker

[+] Building 40.0s (11/20)
 => [internal] load build definition from Dockerfile                                                                                 0.0s
 => => transferring dockerfile: 1.76kB                                                                                               0.0s
 => [internal] load .dockerignore                                                                                                    0.0s
 => => transferring context: 2B                                                                                                      0.0s
 => [internal] load metadata for docker.io/library/debian:bullseye                                                                   0.4s
 => [internal] load build context                                                                                                    0.0s
 => => transferring context: 1.08kB                                                                                                  0.0s
 => CACHED [ 1/16] FROM docker.io/library/debian:bullseye@sha256:10b622c6cf6daa0a295be74c0e412ed20e10f91ae4c6f3ce6ff0c9c04f77cbf6    0.0s
 => [ 2/16] RUN apt-get update && apt-get install -y     wget     curl     lsb-release     software-properties-common     gnupg     28.1s
 => [ 3/16] RUN cd /opt &&     wget https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-14/wasi-sdk-14.0-linux.tar.g  4.4s
 => [ 4/16] RUN touch /opt/wasi-sdk-14.0/bin/wasm32-wasi-readelf &&     chmod +x /opt/wasi-sdk-14.0/bin/wasm32-wasi-readelf          0.2s
 => [ 5/16] RUN cd /tmp &&     wget http://ftp.gnu.org/gnu/autoconf/autoconf-2.71.tar.gz &&     tar zxvf autoconf-*.tar.gz &&     c  3.2s
 => [ 6/16] RUN cd /tmp &&     wget http://ftp.gnu.org/gnu/automake/automake-1.16.tar.gz &&     tar zxvf automake-*.tar.gz &&     c  3.1s
 => ERROR [ 7/16] RUN curl https://wasmtime.dev/install.sh -sSf | bash                                                               0.5s
------
 > [ 7/16] RUN curl https://wasmtime.dev/install.sh -sSf | bash:
#10 0.469 Error: Sorry! Wasmtime currently only provides pre-built binaries for x86_64 architectures.
#10 0.469
------
executor failed running [/bin/sh -c curl https://wasmtime.dev/install.sh -sSf | bash]: exit code: 1

I am on a Mac M1. Docker uses ARM64 as the arch on an M1.

kesmit13 commented 2 years ago

Possibly the solution is to use package libc6-dev rather than libc6-dev-i386? The wasmtime binary issue might be trickier to fix. Technically it isn't needed to do the build, but it's installed for convenience to run Python once it is built.

technosophos commented 2 years ago

I tried changing all the things over to using ARM, but there were just too many little nuances. So I went the opposite route and changed the first line of the Dockerfile to this:

FROM --platform=linux/amd64 debian:bullseye

And then re-running the docker build, followed by docker run with the --platform flag:

$ docker run -it --rm -v $(pwd):$(pwd) --platform linux/amd64 -w $(pwd) wasi-build:latest bash
root@e415ff3c1687:/Users/technosophos/Code/Python/python-wasi# ./run.sh
checking for git... found
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu

This got me all the way to where it runs config to build wasi-python. That errored out with this message:

configure: WARNING: unrecognized options: --with-freeze-module
configure: loading site script /Users/technosophos/Code/Python/python-wasi/config.site
checking for git... found
checking build system type... x86_64-pc-linux-gnu
checking host system type... wasm32-unknown-wasi
checking for --with-build-python... /Users/technosophos/Code/Python/python-wasi/cpython/inst/3.11/bin/python3.11
checking for Python interpreter freezing... /Users/technosophos/Code/Python/python-wasi/cpython/inst/3.11/bin/python3.11
checking for python3.11... (cached) /Users/technosophos/Code/Python/python-wasi/cpython/inst/3.11/bin/python3.11
checking Python for regen version... Python 3.11.0a5+
checking for wasm32-wasi-pkg-config... no
checking for pkg-config... no
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... "wasi"
checking for wasm32-wasi-gcc... clang --target=wasm32-wasi
checking whether the C compiler works... no
configure: error: in `/Users/technosophos/Code/Python/python-wasi/cpython':
configure: error: C compiler cannot create executables
See `config.log' for more details
ERROR: No Python build was found.

I'm gonna let it build once more, and if it fails again, I'll stop until tomorrow.

technosophos commented 2 years ago

Alright, well... after fixing one of my own mistakes (something I had mistakenly commented out), I did get to the point where I could kick off the build. But after four tries -- crashing Docker itself every time -- I have given up.

clang --target=wasm32-wasi -c -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -DPy_BUILD_CORE -o Python/dup2.o Python/dup2.c
clang --target=wasm32-wasi -c -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -DPy_BUILD_CORE -o Modules/config.o Modules/config.c
clang --target=wasm32-wasi -c -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -DPy_BUILD_CORE -o Modules/main.o Modules/main.c
clang --target=wasm32-wasi -c -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -DPy_BUILD_CORE -o Modules/gcmodule.o Modules/gcmodule.c
clang --target=wasm32-wasi -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden  -I./Include/internal  -I. -I./Include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include  -DPy_BUILD_CORE_BUILTIN  -c ./Modules/arraymodule.c -o Modules/arraymodule.o
PLEASE submit a bug report to https://bugs.llvm.org/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.  Program arguments: /opt/wasi-sdk/bin/clang --target=wasm32-wasi -c -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -std=c99 -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wstrict-prototypes -Werror=implicit-function-declaration -fvisibility=hidden -I./Include/internal -I. -I./Include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystem /Users/technosophos/Code/Python/python-wasi/wasix/include -g -D_WASI_EMULATED_GETPID -D_WASI_EMULATED_SIGNAL -D_WASI_EMULATED_PROCESS_CLOCKS -I/opt/include -I/Users/technosophos/Code/Python/python-wasi/wasix/include -isystERRO[0468] error waiting for container: EOF

(Note that that list error waiting for container: EOF hung the Docker daemon itself, and I have to kill/restart each time. I suspect a bug in Docker's amd64 support on M1s)

Any chance you can cut an alpha release of the project and post a binary on the releases page?

kesmit13 commented 2 years ago

It is possible to run the run.sh script directly on a machine that has WASI SDK installed. I'm trying that now because some of the tools haven't been released on linux-aarch64 yet which makes Docker on M1 more challenging. I can post a binary for now.

technosophos commented 2 years ago

Oh, I should give that a try. I think I'd probably just have to change the host target on the cross compile line to use whatever Apple's architecture triplet is.

kesmit13 commented 2 years ago

Yes, there will be a few modifications that have to be made, but they should be minor.

kesmit13 commented 2 years ago

I pushed some changes that make it easier to build on Mac. There are still some issues in getting that work because the Mac release of WASI SDK is not for M1 either. I had to brew install llvm, autoconf, and automake to get the builds working without the bundled clang in WASI SDK.

technosophos commented 2 years ago

Yeah, I have had to build wasi-sdk in the past for exactly that reason. I will give things a try again later this afternoon. Thank you!

technosophos commented 2 years ago

We got it all compiled today, though I'd have to defer to @adamreese on how exactly he got the last few steps working.

zifeo commented 1 year ago

Hey @technosophos @adamreese, are you still able to build for M1? I managed to catch up with the above, yet it is not clear how to overcome that error (tried both with sdk 15 and sdk 19):

wasm-ld: error: unable to find library -lwasix
wasm-ld: error: unable to find library -lwasi_vfs
aichiyuya commented 10 months ago

Hey @technosophos @adamreese, are you still able to build for M1? I managed to catch up with the above, yet it is not clear how to overcome that error (tried both with sdk 15 and sdk 19):

wasm-ld: error: unable to find library -lwasix
wasm-ld: error: unable to find library -lwasi_vfs

hello, i am having the same issue, you can find them there: https://github.com/singlestore-labs/wasix.git https://github.com/kateinoigakukun/wasi-vfs.git