search

zyedidia / micro

A modern and intuitive terminal-based text editor
https://micro-editor.github.io
MIT License
25.19k stars 1.18k forks source link

Static builds on MacOS ARM64 (Apple Silicon aka M1/M2) don't seem to work as expected #2812

Open joshuataylor opened 1 year ago

joshuataylor commented 1 year ago

Description of the problem or steps to reproduce

Building on MacOS ARM64 does not work with static compilation.

To Reproduce Steps to reproduce the behavior:

  1. Use MacOS ARM64
  2. make

Error:

go build -trimpath -ldflags "-s -w -X github.com/zyedidia/micro/v2/internal/util.Version=2.0.12-dev.55 -X github.com/zyedidia/micro/v2/internal/util.CommitHash=651a3010 -X 'github.com/zyedidia/micro/v2/internal/util.CompileDate=April 30, 2023' -linkmode external -extldflags -Wl,-sectcreate,__TEXT,__info_plist,/tmp/micro-info.plist" ./cmd/micro
# github.com/zyedidia/micro/v2/cmd/micro
loadinternal: cannot find runtime/cgo

Hi!

I've been trying to build micro on my M1 (also called Apple Silicon, the newer models are M2.. it's really just AArch64/ARM64), and was getting super obtuse/misleading errors when using the Makefile.

This resulted in researching the fun that is ARM64 MacOS and static linking with Go. I haven't played with Go much, but I have played with llvm etc to know where to dig into why it's not working.. luckily I like deep diving into random topics, and tonights was about static linking madness with MacOS.

Here's what I get when trying to run make build-quick on Darwin ARM64:

$ go version
go version go1.20.3 darwin/arm64

make build-quick

go build -trimpath -ldflags "-s -w -X github.com/zyedidia/micro/v2/internal/util.Version=2.0.12-dev.55 -X github.com/zyedidia/micro/v2/internal/util.CommitHash=651a3010 -X 'github.com/zyedidia/micro/v2/internal/util.CompileDate=April 30, 2023' -linkmode external -extldflags -Wl,-sectcreate,__TEXT,__info_plist,/tmp/micro-info.plist" ./cmd/micro
# github.com/zyedidia/micro/v2/cmd/micro
loadinternal: cannot find runtime/cgo

Confusing error message, as it leads to so many different answers! This works fine when building on a Linux host, as the builds there aren't static:

GOOS=linux GOARCH=amd64 go generate ./runtime
go build -trimpath -ldflags "-s -w -X github.com/zyedidia/micro/v2/internal/util.Version=2.0.12-dev.55 -X github.com/zyedidia/micro/v2/internal/util.CommitHash=651a3010 -X 'github.com/zyedidia/micro/v2/internal/util.CompileDate=April 30, 2023' " ./cmd/micro

That led me down a rabbit hole, and trying to figure out what's going on. Here's a few things I understand:

  1. Code signing is different with Darwin on ARM64. Linkers such as zld have had issues, and other binaries would just return an error on compilation. It's confusing when it happens. See Issue #42684. This is kind of related to this issue (especially when starting to dig), but I don't think this is the actual issue.

  2. There is a fantastic blog post about static compilation with MacOS ARM64 & Go by Matt Turner, see here. Very informative read and good background information about the issue.

  3. There is a fantastic blog post about static compilation in C on MacOS, which should give you more of a background: https://michaelspangler.io/posts/statically-linking-on-macos.html

tldr; Static linking on MacOS can be painful, especially when it references a library that isn't static.

So trying to compile statically ON MacOS ARM64 doesn't work, as it tries to static compile, where as other operating systems don't when cross-compiling.

So yeah, it's tricky. It seems that static compiling on ARM64 MacOS is a bit of a mess, and you can't just static link certain things. :shrug:.

Fiddling with the flags in tools/info.plist gives me this error, which is closer to the actual issue..

/opt/homebrew/Cellar/go/1.20.3/libexec/pkg/tool/darwin_arm64/link: running cc failed: exit status 1
ld: library not found for -lcrt0.o
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Apparently this wasn't supported by gcc shipped by Apple.

This option will not work on Mac OS X unless all libraries (including libgcc.a) have also been compiled with -static. Since neither a static version of libSystem.dylib nor crt0.o are provided, this option is not useful to most people.

I kept trying both the homebrew version of go and the "official" version to see if they return different errors, however they both result in the same errors.

Development builds vs "production builds"

Usually when you are developing vs when you actually want a "production build", you're in the "Write code" -> "Compile" -> "Test change" feedback loop, building without static linking is fine, as this would increase compilation time anyway, right?

When would you want a static build?

This would be for someone who just wants to download and use micro on Darwin ARM64 and not worry about something weird breaking. This is a good thing.. if it worked :-).

Relevant Go issues for this

https://go-review.googlesource.com/c/go/+/477839

So what's the fix?

I remember reading a while ago that you can add which libraries can be static, and which can be dynamic. For MacOS, this should work.

Which frameworks/libraries do we need to add?

Great post here: https://stackoverflow.com/questions/844819/how-to-static-link-on-os-x

On my MBA M1, running Ventura 13.3, if I remove the static linking part:

otool -L micro

micro:
    /usr/lib/libresolv.9.dylib (compatibility version 1.0.0, current version 1.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1971.0.0)
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 60420.101.2)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1319.100.3)

So IMHO, playing with flags like -lresolv etc in tools/info-plist.go is the way to go, then you can define which libraries can be static and which ones are dynamic.

zyedidia commented 1 year ago

I’m a bit confused, are you saying that the Darwin arm64 build doesn’t compile on Darwin arm64? That is surprising since we have nightly binaries for Darwin arm64 that are cross compiled and work fine. If you use CGO_ENABLED=0 make does that work?

joshuataylor commented 1 year ago

Yeah sorry, I also added my notes/ramblings as there is a lot of conflicting/outdated information out there.

I've tried CGO_ENABLED=0, and it doesn't change anything. I believe one of the flags is overriding it.

How are the nightly binaries being built? Is it on a MacOS ARM64 builder, or a generic runner that builds everything? If it's the second, yes that will work as it doesn't hit the logic in info-plist.go, where it adds extra flags for MacOS.

zyedidia commented 1 year ago

I see. The nightly binaries are cross-compiled by a Linux machine. Perhaps those extra flags aren't necessary or should be amended, I'll look into it at the next opportunity I have to use an M1 Mac.

joshuataylor commented 1 year ago

Thanks!

I also figured out how to get builds to work with static linking -- use Go 1.19. go1.19.8 darwin/arm64 works absolutely fine. It looks like Go 1.20 has quite a few changes around cgo, see the release notes.

I included my notes, as it's a bit confusing when researching.

I think a possible solution would be to change how the static linking is done, as you can include which static libraries you want to include, and make the rest dynamic. I recall that when passing flags to the linker, the the last flag has to static, or the linker tries to be helpful and adds other libraries that might make the build fail. The downside with this approach is that if something micro needs changes, the new library will need to be added.

Also see https://github.com/golang/go/issues/58159 , which I think is related, as it looks like the process has changed with go 1.20.