Build packages for multiple platforms (CGO cross compilation)

[omarkohl]

64 bit

• Linux
• Linux ARM (Raspi)
• Windows
• MacOS
• Docker

[omarkohl]

The biggest difficult here is that due to SQLite I am using CGO and that is not trivially supported by goreleaser.

https://goreleaser.com/cookbooks/cgo-and-crosscompiling/

[omarkohl]

https://zig.news/kristoff/building-sqlite-with-cgo-for-every-os-4cic

https://datastation.multiprocess.io/blog/2022-05-12-sqlite-in-go-with-and-without-cgo.html

https://news.ycombinator.com/item?id=31364166

https://www.bytebase.com/blog/how-to-cross-compile-with-cgo-use-goreleaser-and-github-action

[omarkohl]

Options:

• Option 1: Try goreleaser-cross (Docker container plus other shenanigans to make cross-compilation work)
  • :negative_squared_cross_mark: Badly documented
  • :negative_squared_cross_mark: Seems to involve a lot of work to get the sysroots ready for different platforms. How to keep them up to date?
  • :heavy_check_mark: Keep native SQLite
  • :heavy_check_mark: Keep using Goreleaser
• Option 2: Build different binaries on GitHub runners (linux-latest, macos-latest, windows-latest) as described in the bytebase article above
  • :negative_squared_cross_mark: It would mean building all binaries on GitHub, which I find a little less trustworthy than building them locally and which means tying the project a little more tightly to GitHub
  • :negative_squared_cross_mark: It seems that then publishing the result would only work with Goreleaser Pro, which is too expensive, thereby implying that Goreleaser needs to be abandoned
  • :heavy_check_mark: Keep native SQLite, probably with a rather simple build on the whole
  • Option 3: Abandon CGO based SQLite (https://github.com/mattn/go-sqlite3 with GORM driver gorm.io/driver/sqlite a.k.a. https://github.com/go-gorm/sqlite) in favor or pure Go SQLite (https://gitlab.com/cznic/sqlite with GORM driver https://github.com/glebarez/sqlite)
    • :negative_squared_cross_mark: The pure Go SQLite port is probably much less stable and less tested than famously rock-solid original SQLite. It's also slower (see multiprocess.io link above), albeit that is probably not relevant for this project
    • :heavy_check_mark: Makes building, running tests etc. very easy and accessible to more contributors

[omarkohl]

Concerning "Option 2" above some initial experiments with building on Windows were successful (see: https://github.com/cleodora-forecasting/cleodora/blob/d44d87a24e4a91198eb50e9f24b92a6d8efd6748/.github/workflows/build-windows.yml) The resulting binary could be run on Windows.

The downside is that the build was only for cleosrv and did not include the frontend (additional build time, even though in this case it could be built once and shared between several build jobs) and did not include cleoc and still it took about 5 minutes. Considering that I get 2.000 minutes of GitHub action minutes for free, Windows runners count double and Mac OS runners count 10x, it does not seem feasible to do this. (Source: https://docs.github.com/en/billing/managing-billing-for-github-actions/about-billing-for-github-actions)

https://github.com/msys2/setup-msys2

https://www.msys2.org/

[omarkohl]

Other things that might work:

https://dev.to/kristoff/zig-makes-go-cross-compilation-just-work-29ho

https://github.com/crazy-max/xgo (or https://github.com/techknowlogick/xgo - what's the difference, which one is better?)

[omarkohl]

Using Zig the following works on an Linux amd64 host.

# Linux, statically linked
CGO_ENABLED=1 CC="zig cc -target native-native-musl" CXX="zig cc -target native-native-musl" go build github.com/cleodora-forecasting/cleodora/cleosrv

# Windows
CGO_ENABLED=1 GOOS=windows GOARCH=amd64 CC="zig cc -target x86_64-windows" CXX="zig cc -target x86_64-windows" go build -tags=production github.com/cleodora-forecasting/cleodora/cleosrv

The resulting binaries are executable on the respective platforms.

What fails however is the build for MacOS:

CGO_ENABLED=1 GOOS=darwin GOARCH=amd64 CC="zig cc -target x86_64-macos" CXX="zig cc -target x86_64-macos" go build -tags=production github.com/cleodora-forecasting/cleodora/cleosrv

/usr/local/go/pkg/tool/linux_amd64/link: running zig failed: exit status 1
warning: unsupported linker arg: -no_pie
warning(link): framework not found for '-framework CoreFoundation'
warning(link): framework not found for '-framework Security'
warning(link): Framework search paths:
error(link): undefined reference to symbol '_SecTrustSetVerifyDate'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_SecTrustGetCertificateCount'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_SecTrustGetCertificateAtIndex'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_SecTrustEvaluateWithError'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_SecTrustCreateWithCertificates'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_SecPolicyCreateSSL'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_SecCertificateCreateWithData'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_SecCertificateCopyData'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFStringCreateWithBytes'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFStringCreateExternalRepresentation'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFRelease'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFErrorCopyDescription'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFDateCreate'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFDataGetLength'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFDataGetBytePtr'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFDataCreate'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFArrayGetValueAtIndex'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFArrayGetCount'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFArrayCreateMutable'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error(link): undefined reference to symbol '_CFArrayAppendValue'
error(link):   first referenced in '/tmp/go-link-3595762382/go.o'
error: UndefinedSymbolReference

This is documented (https://zig.news/kristoff/building-sqlite-with-cgo-for-every-os-4cic). The solution is using the actual CoreFoundation libs from a real Mac, which I don't have access to.

Given that Zig therefore does not fully solve the problem, it's questionable whether it's worth including this additional dependency. It was surprising how well it works for Windows however.

[omarkohl]

https://dh1tw.de/2019/12/cross-compiling-golang-cgo-projects/

https://github.com/dh1tw/remoteAudio/blob/29928cc5ef7339d6ca1733dc22c74ab8b1b50e9c/.github/workflows/build.yml

https://github.com/dh1tw/remoteAudio-xcompile/

[omarkohl]

I have decided to go with "Option 3" for now, but I will leave this open and hope to get this working in another way. The remoteAudio project above looks quite promising.