Control over what version of `glibc` is used

[jhwz]

I currently use this to cross compile to a few platforms, and it works great! One issue I'm having is the resulting binary is always built with a recent (ish) version of glibc and some of our servers are running an older version of glibc, meaning they can't be run in the native environment.

I understand this package is already doing a lot and this might be out of scope, I'm not entirely sure what needs to change but I'd imagine it has something to do with the Dockerfile using bullseye and not an older environment like stretch? It would be awesome if you were able to choose which environment to use!

[troian]

Hi @jhwz I think the sysroot is what you are looking for 😉 the example is using custom sysroot with all custom libs including glibc.

Let me know if you need any more help with this topic.

[jhwz]

I was coming to that conclusion!

I tried it out and had a few issues with the compiler not being able to find libgcc_so.1:

# runtime/cgo
/usr/bin/ld: cannot find libgcc_s.so.1
collect2: error: ld returned 1 exit status

To extract the sysroot I used the script given, but looking at the custom sysroot repo you linked the sysroot has symlinks that I don't have so there's a chance I'm getting something wrong.. Will keep having a play when I have time!

[troian]

Yeah, symlinks are the hard part when they are absolute. You may need to tinker with PKG_CONFIG_PATH check here and CGO_C(XX)FLAGS.

Should be pretty much it. We've done it on different sorts of projects

[jhwz]

Ahh I had briefly looked at that but didn't know what the PKG_CONFIG_PATH was (this is all out of my comfort zone). Cheers, I'll give that a crack

[troian]

See if the first answer from this thread improves visibility.

I'll close the ticket for now. Feel free to reopen if the issue needs a response from us.

[jhwz]

So interestingly, even after setting the sysroots and setting the PKG_CONFIG_PATH to all of the pkgroots in my clone, the build won't run on the other machine:

./mybinary: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.28' not found (required by ./mybinary)

is that something to do with the version of gcc I'm compiling with?

[troian]

it still seems to link with host glibc. did you mount sysroot to the docker container? this is how it is done in the example I referenced above. try to use linker flag -Wl,-dynamic-linker,<libc path>. with example above it will be something like this -Wl,-dynamic-linker,/lib)/ld-linux.so.2. but honestly, it is unnecessary with modern gcc

also, it is good to enable linker verbose mode to see where it is trying to find libc. -Wl,--verbose is for it.

[jhwz]

Sorry I should give you more context with the commands I'm running, the docker command looks like:

docker run --rm \
    --privileged \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v `pwd`:/build \
    -w /build \
    -v /home/joel/Desktop/hurricane-sysroot:/sysroots/linux_amd64 \
    goreleaser/goreleaser-cross --rm-dist --snapshot

and then in the .goreleaser.yml I have:

  - main: ./cmd/mybinary
    id: linux_amd_build
    binary: mybinary
    env:
      - CGO_ENABLED=1
      - CC=gcc
      - CGO_FLAGS=--sysroot=/sysroots/linux_amd64
      - CGO_LDFLAGS=--sysroot=/sysroots/linux_amd64
      - PKG_CONFIG_SYSROOT_DIR="/sysroots/linux_amd64"
      - PKG_CONFIG_PATH="/sysroots/linux_amd64/usr/lib/x86_64-linux-gnu/pkgconfig:/sysroots/linux_amd64/usr/lib/pkgconfig"
    targets: [linux_amd64]
    tags: [release]

I think it is finding the /sysroots, when I put in the incorrect path for the CGO_FLAGS it complains that it cannot find the correct files, I'll try with the linker flags

[jhwz]

Well you're right, it isn't linking against the correct libraries but I have no idea why! If I try build myself in the container doing:

export PKG_CONFIG_SYSROOT_DIR=/sysroots/linux_amd64
export PKG_CONFIG_PATH="/sysroots/linux_amd64/usr/lib/x86_64-linux-gnu/pkgconfig:/sysroots/linux_amd64/usr/lib/pkgconfig"
export CGO_ENABLED=1
export CGO_FLAGS=--sysroot=/sysroots/linux_amd64export CGO_LDFLAGS="-Wl,--verbosesysroot=/sysroots/linux_amd64"
go build -o mybin ./

then I get a bunch of logs like:

attempt to open /usr/lib/gcc/x86_64-linux-gnu/10/libgcc.so failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/10/libgcc.a succeeded
/usr/lib/gcc/x86_64-linux-gnu/10/libgcc.a
attempt to open /usr/lib/gcc/x86_64-linux-gnu/10/libgcc_s.so succeeded
opened script file /usr/lib/gcc/x86_64-linux-gnu/10/libgcc_s.so
/usr/lib/gcc/x86_64-linux-gnu/10/libgcc_s.so
opened script file /usr/lib/gcc/x86_64-linux-gnu/10/libgcc_s.so
attempt to open /usr/lib/gcc/x86_64-linux-gnu/10/libgcc_s.so.1 failed
attempt to open libgcc_s.so.1 failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/10/libgcc_s.so.1 failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/10/../../../x86_64-linux-gnu/libgcc_s.so.1 failed
attempt to open /usr/lib/gcc/x86_64-linux-gnu/10/../../../../lib/libgcc_s.so.1 failed
attempt to open /lib/x86_64-linux-gnu/libgcc_s.so.1 succeeded

so it's clearly looking in the wrong place. I don't understand what the PKG_CONFIG_SYSROOT_DIR variable is actually doing which doesn't help either.

I also tried building without the CGO_FLAGS and CGO_LDFLAGS to no avail

[troian]

it should be CGO_CFLAGS i think.

[jhwz]

Ahh that worked! Thanks for your time, really appreciate it

[troian]

Apparently, example I gave you had this typo, sorry for the confusion. Glad tho issue has been resolved.

[pskrbasu]

Hey @troian I have been hit with the same issue. I have a simple cgo project that I want to run on my Amazon Linux 2 server which has a lower glibc version(2.26).

I tried using goreleaser cross without the sysroot approach as given in the example. I got this error:

./cgo: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by ./cgo)
./cgo: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by ./cgo)

Then I rsynced the sysroot directory using the script like this: ./sysroot-rsync.sh -i ~/keys/key.pem ec2-user@xxx.xxx.xxx.xxx:/usr/ ~/src/cgo-cross-compilation/sysroot

I rsynced the whole /usr directory from the Amazon Linux EC2 instance. Is this correct? (you can find the rsynced sysroot contents here).

Then I modified my Makefile and Goreleaser.yml file as per the example and built the binary.

But running it again on the Amazon Linux EC2 server gave the same error(which was unexpected):

./cgo: /lib64/libm.so.6: version `GLIBC_2.29' not found (required by ./cgo)
./cgo: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.26' not found (required by ./cgo)

The sysroot libm.so.6 file has a version of 2.26 (I verified), so I expected the built binary to work this time.

$ strings sysroot/lib64/libm.so.6 | grep GLIBC_2.26
GLIBC_2.26

Is there anything wrong I am doing? Do I need to do anything differently(the rysncing of sysroot)?

Any help will be greatly appreciated.

Thanks

cc @jhwz

[jhwz]

Not sure sorry but from the error it looks like the Go runtime (cgo) itself requires at least version GLIBC_2.29. You'll either need to upgrade your server or downgrade your version of Go.

[troian]

@pskrbasu add this flag to CGO_LDFLAGS -Wl,--verbose output will be rather huge, what you'll need to look for is what libs linker is picking up