Build Linux binaries with musl

igor-hnizdo commented 2 years ago

When I tried to build with "normal" glibc, I got weird errors on old/new glibc.

Instead of building with glibc, which limits us with compatibility and weird issues, let's build with musl, which will always produce 100% static library.

(Musl is a libc alternative. The good thing is musl is statically linked and can cross-compile relatively easily.)

The actual hard work is building all these musl cross-compilers; luckily, Rust people already did that for Rust :) we can steal that and reuse.

Note that this all works on both AMD and ARM hosts, as both the musl cross-compilers images and the go images are published for both.

igor-hnizdo commented 2 years ago

Converting it to draft. I think building static library with glibc is also possible, and at least we will depend only on the elastic repo, and not on the musl repo.

But let me try first. This still does work.

prusnak commented 2 years ago

I am fine with building the static binaries with musl. I think it's better than trying to use static with glibc.

igor-hnizdo commented 2 years ago

Yeah I tried to use static and libc and there were some weird issues. OK, putting it back to in progress.

tsusanka commented 2 years ago

Btw @igor-hnizdo why don't we use golang-crossbuild again for linux? We can't do that?

Oh or that's what you are writing in the original post? That it uses glibc which causes troubles?

igor-hnizdo commented 2 years ago

Yes. I tried that and I got some incompatibilities with the old glibc (I am not sure why, I got some random error when building); and the newer glibc might be incompatible with older Linux distros... maybe?

I did not want to debug this and "just" put there musl, which always builds all statically.

Unfortunately golang-crossbuild does not support musl (I have opened a issue there now)

https://github.com/elastic/golang-crossbuild/issues/230

igor-hnizdo commented 2 years ago

Well, to do some investigation.

golang-crossbuild with base images debian 7 and 8 does not work, because they have too old gcc.
golang-crossbuild with base image debian 9 does work, but that uses glibc 2.24
if we see the list of https://gist.github.com/wagenet/35adca1a032cec2999d47b6c40aa45b1 , there are lot of TLS distos with older glibc
it probably worked with xgo before, as that install gcc-5 manually?
we can install gcc 5 manually in the golang-crossbuild ourselves, but then we are already doing some patching... let's just use musl, which will always work :) and will save us distro-compatibility troubles long term, IMO

tsusanka commented 2 years ago

The docker runs fine, I can't test the binary as I have only macOS and NixOS available, but I think we can merge.

@mroz22 if you would like to feel free to test the build on Linux:

cd release/linux/
make clean all
build/trezord-linux-386

and see if it launches okay.

mroz22 commented 2 years ago

make clean all produced

+ CGO_ENABLED=1 CC=/usr/local/musl/bin/aarch64-unknown-linux-musl-gcc GOARCH=arm64 go build --ldflags -linkmode external -extldflags "-static" -extld /usr/local/musl/bin/aarch64-unknown-linux-musl-gcc -o release/linux/build/trezord-linux-arm64
go: downloading gopkg.in/natefinch/lumberjack.v2 v2.0.0
go: downloading github.com/gorilla/handlers v1.5.1
go: downloading github.com/gorilla/mux v1.8.0
go: downloading github.com/gorilla/csrf v1.7.0
go: downloading github.com/felixge/httpsnoop v1.0.1
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/gorilla/securecookie v1.1.1
error obtaining VCS status: exit status 128
    Use -buildvcs=false to disable VCS stamping.
make: *** [Makefile:18: .binary] Error 1

and it did not finish the build

igor-hnizdo commented 2 years ago

huh that's weird. I will look at that.

igor-hnizdo commented 2 years ago

It seems to be something with the git directory ownership and the way go does the magic stamping. (And how Docker on linux does users.)

https://github.com/golang/go/issues/51253

I will try this on Linux computer to test and see

igor-hnizdo commented 2 years ago

Yeah. New git (released this april) adds a security measure, that dis-allows git traversal if the user is a different user.

It's a mitigation against CVE-2022-24765

https://nvd.nist.gov/vuln/detail/cve-2022-24765

This vulnerability affects users working on multi-user machines, where untrusted parties have write access to the same hard disk. Those untrusted parties could create the folder C:\.git, which would be picked up by Git operations run supposedly outside a repository while searching for a Git directory. Git would then respect any config in said Git directory.

And the user inside the docker is root, a different user than who owns the outside repo. And go then calls git which then errors.

We don't see this in the crossbuild repos, as they use old git. And we don't see it on macOS, as on macOS, docker pretends all shared files are from root, or something like that. (unlike on Linux, where it's just a layer over cgroups or some other linux magic, on macos it has all the virtual machine stuff added.)

The easiest solution is to add safe.directory to the build.sh, which I will now do.

igor-hnizdo commented 2 years ago

The fix here

https://github.com/trezor/trezord-go/pull/267

trezor / trezord-go

Build Linux binaries with musl #263