[regression] cgo/boringssl builds not working on alpine 3.19

[ashishb-solo]

Description

Builds that use cgo and boringssl appear to have regressed from alpine 3.18 to 3.19 - builds that worked successfully before now appear to provide the following linking error:

# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: /tmp/go-link-2280080120/000032.o: in function `BIO_new_file':
(.text+0xcb1dc): undefined reference to `fopen64'
/usr/lib/gcc/x86_64-alpine-linux-musl/13.2.1/../../../../x86_64-alpine-linux-musl/bin/ld: /tmp/go-link-2280080120/000032.o: in function `BIO_rw_filename':
(.text+0xcb60d): undefined reference to `fopen64'
collect2: error: ld returned 1 exit status

Significance

This issue is significant, because we believe that this is required for anyone using boringssl cryptography for FIPS compliance.

Reproduction

I was able to reproduce this using a minimal dockerfile - the example provided below should fail to build with the above error message:

FROM golang:alpine3.19

RUN mkdir /go/src/example
WORKDIR /go/src/example
RUN apk add gcc musl-dev libc6-compat
RUN wget -O main.go https://raw.githubusercontent.com/kubernetes/client-go/master/examples/out-of-cluster-client-configuration/main.go
RUN go mod init example && go mod tidy
RUN GOEXPERIMENT=boringcrypto CGO_ENABLED=1 go build main.go

Attempting to build this docker file results in the linker error that I quoted above. Changing the version tag from golang:alpine3.19 to golang:alpine3.18 causes the build to complete successfully.

Additional Context

We've dug a little bit into this issue and have found this interesting bit from the Alpine 3.19 release notes:

GNU C Library Compatibility Layer

libc6-compat was removed in favor of gcompat from Adélie Linux.

A quick scan of the libraries on both systems appears to indicate that the fopen64 symbol has been removed from libc - see below. The first command scans libraries for fopen64 on alpine 3.18, and the second scans does the same on alpine 3.19, but fopen64 only appears on 3.18:

ashish@localhost ~ $ docker run --rm -it alpine:3.18 find /lib -exec grep fopen {} +
/lib/libc.musl-x86_64.so.1:fopen
/lib/libc.musl-x86_64.so.1:fopen64
/lib/libc.musl-x86_64.so.1:fopencookie
/lib/ld-musl-x86_64.so.1:fopen
/lib/ld-musl-x86_64.so.1:fopen64
/lib/ld-musl-x86_64.so.1:fopencookie
/lib/libapk.so.2.14.0:fopen
/lib/libcrypto.so.3:fopen
/lib/libcrypto.so.3:bad fopen mode
/lib/libcrypto.so.3:calling fopen(%s, %s)

ashish@localhost ~ $ docker run --rm -it alpine:3.19 find /lib -exec grep fopen {} +
Unable to find image 'alpine:3.19' locally
3.19: Pulling from library/alpine
661ff4d9561e: Already exists 
Digest: sha256:51b67269f354137895d43f3b3d810bfacd3945438e94dc5ac55fdac340352f48
Status: Downloaded newer image for alpine:3.19
/lib/libc.musl-x86_64.so.1:fopen
/lib/libc.musl-x86_64.so.1:fopencookie
/lib/libc.musl-x86_64.so.1:fopen
/lib/ld-musl-x86_64.so.1:fopen
/lib/ld-musl-x86_64.so.1:fopencookie
/lib/ld-musl-x86_64.so.1:fopen
/lib/libapk.so.2.14.0:fopen
/lib/libcrypto.so.3:fopen
/lib/libcrypto.so.3:bad fopen mode
/lib/libcrypto.so.3:calling fopen(%s, %s)

Happy to help in any way that I can in resolving this, but am not completely sure where exactly to start. fopen64 appears to have been removed for good from Alpine, so it's not immediately clear where this ought to be resolved

[tianon]

I think the upstream references to fopen64 (at least, the only ones I can actually find) are all in the pre-compiled src/crypto/internal/boring/syso/goboringcrypto_linux_amd64.syso (and src/crypto/internal/boring/syso/goboringcrypto_linux_arm64.syso) files, so I think those would probably need to be recompiled to fix this? Perhaps via one of the "build" scripts in src/crypto/internal/boring/ ?

(Either way, this is definitely an upstream bug, although one I don't know whether they'll prioritize very highly since Alpine/musl is not officially supported by the Go project. https://github.com/golang/go/issues/19938)

[ashishb-solo]

Thanks for your response @tianon. Since it seems unlikely that upstream Go/Boringssl will fix this, then perhaps there's a chance that this can get resolved in upstream Alpine since the library is supposed to be provided by the system. I went ahead and created an issue for it there: https://gitlab.alpinelinux.org/alpine/aports/-/issues/15573

[ashishb-solo]

It seems that upstream Alpine's plan is to recompile the problematic libraries, ie. src/crypto/internal/boring/syso/goboringcrypto_linux_amd64.syso. This should fix upstream Alpine, however it will probably not resolve the issue in this docker image because the image installs go by downloading binaries from the upstream go projects and not the alpine package repositories (right?).

So I've opened an issue with golang upstream here and if it gets addressed there, then it should fix both Alpine and this image (hopefully :-P)...

[tianon]

https://github.com/golang/go/issues/64698#issuecomment-1856649087 is pretty darn definitive :sweat_smile: :heart: