GLIBC Errors when building

[bketelsen]

When building on linux, I'm getting GLIBC errors unless I'm running on a very recent Ubuntu version. (> 20.04?)

Reproduction dockerfiles for wagi (works fine) and spin(GLIBC errors) at this gist:

https://gist.github.com/bketelsen/2e2c82fb15edec676ae5194130fa7a55

Spin @ b39acc384f66a9c22aad10b2a05b6541ed71383e Wagi @ 0.8.1

Reproduced on physical machine and docker.

[radu-matei]

Relevant part of the error from the gist.

It seems to be part of the build script (build.rs):

error: failed to run custom build command for `spin-cli v0.1.0 (/usr/src/spin)`

Caused by:
  process didn't exit successfully: `/usr/src/spin/target/release/build/spin-cli-591d128a990b5f46/build-script-build` (exit status: 1)
  --- stderr
  /usr/src/spin/target/release/build/spin-cli-591d128a990b5f46/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.29' not found (required by /usr/src/spin/target/release/build/spin-cli-591d128a990b5f46/build-script-build)
  /usr/src/spin/target/release/build/spin-cli-591d128a990b5f46/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /usr/src/spin/target/release/build/spin-cli-591d128a990b5f46/build-script-build)
  /usr/src/spin/target/release/build/spin-cli-591d128a990b5f46/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/src/spin/target/release/build/spin-cli-591d128a990b5f46/build-script-build)
  /usr/src/spin/target/release/build/spin-cli-591d128a990b5f46/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/src/spin/target/release/build/spin-cli-591d128a990b5f46/build-script-build)
warning: build failed, waiting for other jobs to finish...
error: build failed
make: *** [Makefile:6: build] Error 101
The command '/bin/sh -c make build' returned a non-zero code: 2

[lann]

I am trying to minimally reproduce, but on rust:1.59-slim-buster I am able to cargo build --release spin @ https://github.com/fermyon/spin/commit/b39acc384f66a9c22aad10b2a05b6541ed71383e without an error.

Could you clarify what is running with docker build . -f Dockerfile.base? I don't see that file in the gist.

Edit: I guess in particular are there any local changes to the spin repo from that COPY . .?

[lann]

~It just occurred to me that those binaryen bins are in the PATH. I suspect those are being picked up by something in the rust build process which could easily cause a problem~

[lann]

Not sure if its relevant, but it looks like the docker image rust:1.59-slim-buster has been updated. My image ID is 3c48ac985493 where yours was 3ca1d8ed11bf.

[lann]

I made a more rigorous attempt to reproduce (with a totally clean git checkout) but I'm still not seeing this failure.

To rule out a stealth bug fix with that image change could you docker pull rust:1.59-slim-buster and try again?

[bketelsen]

will do!

[lann]

error: failed to run custom build command for spin-cli v0.1.0 (/usr/src/spin)

This is a failure in build.rs, which is part of the cargo build process. That build program compiles some test wasm progams by forking more calls to cargo. I'm unsure whether the error you're seeing is a failure to execute the build program or an error in those forked cargo calls.

If the above doesn't help the next step we could try would be changing the RUN make build line to RUN cargo build --release -vv; the -vv should emit much more detailed logging including some helpful info from the build script.

[bketelsen]

I originally go this error compiling from a recent distro, copying the binary to an older one (older libc/glibc) I'm going to try to replicate that way first.

[bketelsen]

I can't reproduce now. Closing.

[radu-matei]

I think @jdbohrman has seen this on WSL as well. Reopening.

[lann]

Do we have debug logs for the failure?

[radu-matei]

@gmlewis in #345:

When working through the "Taking Spin for a spin" quickstart page, I get this on a Linux Mint 19.3 box:

$ spin up
spin: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.29' not found (required by spin)
spin: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by spin)

$ file `which spin`
/home/glenn/tools/bin/spin: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1f98ec953b04be93667e5222883203df722f05af, for GNU/Linux 3.2.0, with debug_info, not stripped

Can't spin be provided as a statically-linked binary executable?

[radu-matei]

Currently trying to build a statically linked executable with:

RUSTFLAGS="-C target-feature=+crt-static" cargo build --release`

[itowlson]

Ooh, this might come in useful for embedding scenarios too - one of the things I ran into linking with a C main was missing imports. (Although I suppose then we might get conflicts between C or other host language wanting to import glibc and the Rust lib already having it sigh.)

[radu-matei]

So is the command here (https://github.com/fermyon/spin/issues/168#issuecomment-1096842282) generating a statically linked binary, @gmlewis?

(Don't have access to a Linux machine right now, so can't really test this.)

If it is, then we should update the CI bit that builds this on Linux.

[gmlewis]

Oh, I didn't understand you were asking me to try building it. 😁 I had installed from the release tarball. I'll try and get back to you when I'm back at my computer.

[gmlewis]

@radu-matei - here's what I get:

$ RUSTFLAGS="-C target-feature=+crt-static" cargo build --release
error: cannot produce proc-macro for `async-stream-impl v0.3.3` as the target `x86_64-unknown-linux-gnu` does not support these crate types

Note, however, that cargo build --release works fine on my machine, so I'm up and spinning:

$ cargo build --release
...
    Finished release [optimized] target(s) in 2m 31s

$ file target/release/spin
target/release/spin: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=fb5c086ac1befaf9d20ee4d765740f7b0dcbc277, with debug_info, not stripped

$ cd examples/http-rust/
$ spin up
Serving HTTP on address http://127.0.0.1:3000

$ curl -i localhost:3000/hello
HTTP/1.1 200 OK
foo: bar
content-length: 16
date: Tue, 12 Apr 2022 23:54:35 GMT

Hello, Fermyon!

Thank you! :tada:

[radu-matei]

Yaay, glad it works for you!

[radu-matei]

Also, @gmlewis, could you please try out our canary binary from https://github.com/fermyon/spin/releases/tag/canary? Just trying to make sure it's still an issue on main.

Thanks a lot!

[gmlewis]

Also, @gmlewis, could you please try out our canary binary from https://github.com/fermyon/spin/releases/tag/canary? Just trying to make sure it's still an issue on main.

Sorry, it appears to still be an issue on the canary:

$ tar xf ../spin-canary-linux-amd64.tar.gz 
$ ls -l
total 31556
-rw-r--r-- 1 glenn glenn    11602 Apr 13 11:39 LICENSE
-rw-r--r-- 1 glenn glenn     2235 Apr 13 11:39 readme.md
-rwxr-xr-x 1 glenn glenn 32270056 Apr 13 11:39 spin

$ file spin
spin: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=214ac691733930151ae7be783d3064b56e75c1f2, for GNU/Linux 3.2.0, with debug_info, not stripped

$ ./spin up
./spin: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.29' not found (required by ./spin)
./spin: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by ./spin)

[tschneidereit]

It might be useful to do something similar to what wasmtime does: do the builds in a container on as old a distro as we can get away with, to ensure as much compatibility as we can get: https://github.com/bytecodealliance/wasmtime/tree/main/.github/actions/binary-compatible-builds

If you sort out an approach that's better than that and leads to more self-contained builds, we'd love to have that for upstream wasmtime, too 😬

[gmlewis]

Can you use MUSL to generate a statically-linked binary executable in Rust? https://stackoverflow.com/questions/31770604/how-to-generate-statically-linked-executables

[radu-matei]

@gmlewis — I think this is what we tried with https://github.com/fermyon/spin/issues/168#issuecomment-1096842282, it might be worth exploring adding a CI target that builds such a binary.

@tschneidereit — that is a great suggestion. If we don't find a workaround, we will fall back to building in a container (looping in @vdice , as this would change our build process a bit).

[chairwa]

I can static build spin this way:

apt install musl-tools cd /usr/bin ln -s musl-gcc aarch64-linux-musl-gcc

Cargo.toml file: [dependencies] openssl = { version = "*", features = ["vendored"] }

rustup target add aarch64-unknown-linux-musl cargo build --target aarch64-unknown-linux-musl --release

file target/aarch64-unknown-linux-musl/release/spin target/aarch64-unknown-linux-musl/release/spin: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=ce1c2c636ae624747a8a00804a87663d455c27f3, with debug_info, not stripped

[gmlewis]

I can static build spin this way:

Following this recipe and replacing aarch64-unknown-linux-musl with x86_64-unknown-linux-musl (from here ), I was also able to successfully build a statically linked executable:

$ file target/x86_64-unknown-linux-musl/release/spin
target/x86_64-unknown-linux-musl/release/spin: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), statically linked, BuildID[sha1]=b1c9677a6e2dc91261ddbdd064d51943f265ed60, with debug_info, not stripped

Thank you, @chairwa !

[lann]

I'm going to close this issue. We have to pick some glibc version to build against (musl brings its own compatibility quirks), so this will likely always be an issue for someone.

[mikkelhegn]

The above is a consistent issue, when trying to run Spin on dev container images provided by Microsoft - e.g.,

mcr.microsoft.com/devcontainers/javascript-node:1-18-bullseye

Distributor ID: Debian
Description:    Debian GNU/Linux 11 (bullseye)
Release:        11
Codename:       bullseye

spin: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by spin)
spin: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by spin)
spin: /lib/aarch64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by spin)

Ref: https://discord.com/channels/926888690310053918/1146318527402365028

Can we provide an alternate build for this scenario? Or build on an even older OS?

[lann]

That is surprising. Our releases are built on ubuntu 20.04, which is based on bullseye...

[itowlson]

Building on an "even older" OS would involve creating a custom build environment - we are on the oldest Ubuntu available "off the shelf" in GitHub CI.

[tschneidereit]

It might make sense to evaluate using the same approach as Wasmtime. It's possible that we'd run into more problems with that though, because as opposed to Wasmtime we have dependencies that use OpenSSL, which traditionally can be a huge pain.