cmd/go: macOS on arm64 requires codesigning

[FiloSottile]

On the production Apple Silicon machines, Go binaries are killed at start. https://github.com/golang/go/issues/38485#issuecomment-729301832

It looks like all binaries need to be codesigned now, and indeed running codesign -s - on them lets them run correctly.

This stops go test and go run from working, and requires an extra step after go build to get a functional binary.

This also affects the bootstrapped compiler itself.

[FiloSottile]

@cherrymui indeed, binaries produced by the machine's clang are codesigned, in what looks like the same exact way as the output of codesign -s -.

filippo@Filippos-MacBook-Pro tmp % clang hello.c
filippo@Filippos-MacBook-Pro tmp % ./a.out
Hello M1!
filippo@Filippos-MacBook-Pro tmp % codesign -d -v a.out                                 
Executable=/Users/filippo/tmp/a.out
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=510 flags=0x20002(adhoc,linker-signed) hashes=13+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none

[FiloSottile]

Actually, codesign outputs don't have the linker-signed flag.

[cherrymui]

Thanks, @FiloSottile !

Interesting. The C compiler on the DTK doesn't do that...

Could you try one more thing: run clang -v hello.c to see if the C compiler invokes codesign or how it signs the binary? Thanks!

[FiloSottile]

It doesn't look like the linker invocation has anything special, so I assume ld just does that (and @jedisct1 said so on Twitter as well https://twitter.com/jedisct1/status/1328862207715794946).

"/Library/Developer/CommandLineTools/usr/bin/ld" -demangle -lto_library /Library/Developer/CommandLineTools/usr/lib/libLTO.dylib -no_deduplicate -dynamic -arch arm64 -platform_version macos 11.0.0 11.0 -syslibroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -o a.out -L/usr/local/lib /var/folders/jh/3ydm4lxd71s2__g_x4hny6r00000gn/T/hello-580ca7.o -lSystem /Library/Developer/CommandLineTools/usr/lib/clang/12.0.0/lib/darwin/libclang_rt.osx.a

I expect external linking should generate codesigned binaries, but trying `go build -ldflags="-linkmode=external" fails immediately with

# tmp
loadinternal: cannot find runtime/cgo

[cherrymui]

loadinternal: cannot find runtime/cgo

This is not a failure. It prints a message but it should successfully link the binary.

[FiloSottile]

You're right, I do get the binary. Interestingly, it comes out linker-signed but doesn't run, and codesign doesn't work on it.

filippo@Filippos-MacBook-Pro tmp % codesign -d -v tmp                                                       
Executable=/Users/filippo/tmp/tmp
Identifier=a.out
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=7038 flags=0x20002(adhoc,linker-signed) hashes=217+0 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements=none
filippo@Filippos-MacBook-Pro tmp % ./tmp             
zsh: killed     ./tmp
filippo@Filippos-MacBook-Pro tmp % codesign -s - tmp 
tmp: the codesign_allocate helper tool cannot be found or used

[cherrymui]

Thanks!

Interesting... I'll see if it is possible to generate LC_CODE_SIGNATURE in the linker. If that's not possible, maybe shell out codesign.

Any down side for that? If the user wants to sign with a different identity, would that still be possible?

[ianlancetaylor]

I don't understand this. What is the point of code signing, if every execution of clang or the Go compiler produces a code signed binary?

[FiloSottile]

Any down side for that?

I guess the main problem would be with cross-compilation, but anyway I guess adhoc signatures only work on the same machine where they are generated.

This would be a major pain for projects distributing binaries. I wonder how Homebrew deals with it, since by default they install pre-built "Bottles".

What is the point of code signing, if every execution of clang or the Go compiler produces a code signed binary?

I don't know, but I can imagine 1) to normalize the execution path such that it always check signatures unconditionally, simplifying it and 2) to block binaries generated on other machines (and not properly signed by a trusted Developer ID) which before was sort of the role of the quarantine attributes.

[terinjokes]

You're right, I do get the binary. Interestingly, it comes out linker-signed but doesn't run, and codesign doesn't work on it.

My understanding is this is a bug already reported to Apple, see https://github.com/Homebrew/brew/issues/9082#issuecomment-727247739

I wonder how Homebrew deals with it, since by default they install pre-built "Bottles".

Per https://github.com/Homebrew/brew/issues/7857#issuecomment-727561399 I don't think they're close to code signing bottles on ARM yet.

[rolandshoemaker]

2) to block binaries generated on other machines (and not properly signed by a trusted Developer ID) which before was sort of the role of the quarantine attributes.

Looks like binaries aren't required to have a signature linked to a Developer ID if they were signed elsewhere. I was able to apply an ad-hoc signature to a binary on my Intel mac and transfer it to my M1 mac and it ran just fine. Appears like binaries just need a signature, doesn't really seem to matter where it came from.

[fxcoudert]

Homebrew maintainer here:

the codesign_allocate helper tool cannot be found or used

That's indeed a bug in codesign, and Apple is aware

My understanding is this is a bug already reported to Apple, see Homebrew/brew#9082 (comment)

Yes. If this occurs, you need to change that file's inode (so copying it someplace and back will work). Once you've done that, you can sign and it will work. See https://github.com/Homebrew/brew/pull/9102 for details.

I don't think they're close to code signing bottles on ARM yet

We do not distribute bottles yet, because our CI is not yet fully operational, but our codebase is otherwise ready. We apply ad-hoc signatures as part of formula installation: https://github.com/Homebrew/brew/pull/9102

[fxcoudert]

As for go, if Go uses its own linker (or anything other than Apple's ld) then it definitely needs to call codesign -s - on the binaries it produces: executables, shared libraries, etc.

[mislav]

@FiloSottile (and others): thank you for the information! 🙇

Do you know if it is possible to use darwin/amd64 architecture to cross-compile binaries that are able to execute on darwin/arm64, providing that the binaries are explicitly signed?

I am interested in what are the options to distribute binaries for Go projects that work on Apple Silicon without having to actually compile them on Apple Silicon.

[bwesterb]

@mislav Cross-compiling for darwin/arm64 works fine from darwin/amd64 and even linux/amd64. Obviously the resulting binaries are not signed. They will run if signed ad hoc on the same machine they will run. Binaries ad hoc signed on one machine didn't work on another machine for me as well.

[jameshartig]

Binaries ad hoc signed on one machine didn't work on another machine for me.

This comment seems to disagree: https://github.com/golang/go/issues/42684#issuecomment-729346276

I was able to apply an ad-hoc signature to a binary on my Intel mac and transfer it to my M1 mac and it ran just fine.

[bwesterb]

Ok, I tested it again and now cross-signing does work. I might've transferred the wrong file earlier. Whoops.

[FiloSottile]

Codesigning from cmd/link should be possible. https://github.com/isignpy/isign by @neilk does that for iOS.

[networkimprov]

The Zig project is also working on this here: ziglang/zig#7103 (thanks @komuw).

[cherrymui]

Thanks for the pointers. I'll look into them.

[fxcoudert]

Binaries ad hoc signed on one machine didn't work on another machine for me

I can confirm that a linker or other ad hoc signature is sufficient to run the binary (or shared library) on any machine.

[rolandshoemaker]

A quick note from trying to hack together a working toolchain today: shelling out to codesign in the linker works, but any subsequent calls to buildid to update the build ID appears to break the signature (preventing the update produces working binaries). Presumably the signature will need to be done (either magically by the linker with LC_CODE_SIGNATURE or shelling to codesign) as the last step after any changes are made to the object.

[cherrymui]

@rolandshoemaker yeah, I tried that, too, and come to the same conclusion.

I'm not sure whether signing after stamping buildid will cause any issue for the go command's staleness analysis, though.

[marcopeereboom]

This is total crap. Guess Apple does not want developers using their new hardware? Or forcing them to use the entire operating system as a very fancy ssh terminal.

[dmitshur]

@marcopeereboom Please be mindful of Gopher values when communicating in the Go issue tracker. Criticism is welcome when it's constructive and adds new information to the discussion, otherwise it does not help us get to a resolution of this issue.

[FiloSottile]

@kb091412 on Twitter found the official docs, which basically confirm everything we had deduced: https://developer.apple.com/documentation/macos-release-notes/macos-big-sur-11_0_1-universal-apps-release-notes#Code-Signing

New in macOS 11 on Macs with Apple silicon, and starting in macOS Big Sur 11 beta 6, the operating system enforces that any executable must be signed before it’s allowed to run. There isn’t a specific identity requirement for this signature: a simple ad-hoc signature is sufficient. This new behavior doesn’t change the long-established policy that our users and developers can run arbitrary code on their Macs, and is designed to simplify the execution policies on Macs with Apple silicon and enable the system to better detect code modifications. This new policy doesn’t apply to translated x86 binaries running under Rosetta 2, nor does it apply to macOS 11 running on Intel-based platforms.

To reduce the impact on existing development workflows, starting with Xcode 12 beta 4, the toolchain will now automatically sign your executables whenever you build from Xcode, or use command-line utilities such as clang(1) or ld(1). Since this signing mechanism generates signatures directly at link time, and doesn’t cover any resource other than the executable, it is expected to be faster than a traditional codesign(1) invocation. If you use a custom workflow involving tools that modify a binary after linking (e.g. strip or install_name_tool) you might need to manually call codesign(1) as an additional build phase to properly ad-hoc sign your binary. These new signatures are not bound to the specific machine that was used to build the executable, they can be verified on any other system and will be sufficient to comply with the new default code signing requirement on Macs with Apple silicon. However, given that these signatures do not bear any valid identity, binaries signed this way cannot pass through Gatekeeper.

[jrick]

Interesting. Would love to see Go implement this ad-hoc signing directly instead of needing to shell out to codesign, so that cross compiling usable binaries is still possible. Would also be good to use a static key so we can keep reproducibility.

[leo60228]

An open-source ad-hoc signing tool was created for NixOS: https://github.com/thefloweringash/sigtool

[leo60228]

Using it requires Apple's codesign_allocate, but that's open-source as well: https://opensource.apple.com/source/cctools/cctools-949.0.1/misc/codesign_allocate.c.auto.html

[rolandshoemaker]

@cherrymui Oh hm, good point. I think signing after buildid would indeed break the staleness check, since adding the LC_CODE_SIGNATURE section will change the file hash, just as stamping buildid after signing breaks the signature for the same reason.

It seems like perhaps buildid on darwin would need to understand where the LC_CODE_SIGNATURE section is (or would be) in the object in order to create a hash that ignores that section and can be recalculated after the signature is added 😕.

[meme]

Found a workaround that I'm using for now. The AppleMobileFileIntegrity.kext is what handles checking for code signing, and you can see in the Console that AMFI prohibits running:

AMFI: hook..execve() killing pid 9799: Attempt to execute completely unsigned code (must be at least ad-hoc signed).

However there is an undocumented boot argument called amfi_block_unsigned_code which disables this check.

Enter recovery mode by holding down the MacBook power button when the MacBook is powered off and wait until it says that it's loading the settings. Enter settings and open a terminal:

csrutil enable --without nvram

This will leave all security features intact except disable "NVRAM Protections" and "Boot-arg Restrictions". Reboot back into macOS and set the boot argument:

sudo nvram boot-args="amfi_block_unsigned_code=0"

Now reboot once again and you'll be able to run unsigned applications. Note however that a new message will be presented at boot:

AMFI: You have used the amfi_block_unsigned_code boot-arg. This will be removed in a future release.

This will be useful for developing the Go toolchain on Apple Silicon devices and toggling this behaviour.

[kubkon]

Using it requires Apple's codesign_allocate, but that's open-source as well: https://opensource.apple.com/source/cctools/cctools-949.0.1/misc/codesign_allocate.c.auto.html

The good news here is that codesign_allocate is only used to verify the basic MachO layout of the __LINKEDIT hidden sections, and then inject an empty, 0-padded placeholder for the code signature as the very last section, plus a matching LC_CODE_SIGNATURE load command.

[kubkon]

@cherrymui Oh hm, good point. I think signing after buildid would indeed break the staleness check, since adding the LC_CODE_SIGNATURE section will change the file hash, just as stamping buildid after signing breaks the signature for the same reason.

It seems like perhaps buildid on darwin would need to understand where the LC_CODE_SIGNATURE section is (or would be) in the object in order to create a hash that ignores that section and can be recalculated after the signature is added 😕.

Do you think the code signature will always be of the same size? What I inferred about codesign is that it precalculates the size needed and then calls codesign_allocate to preallocate the space in the MachO binary before the actual SuperBlob structure is generated and inserted.

[leo60228]

Based on sigtool's source, it seems like the size of the signature is variable. I'm not a Go developer, but I'd guess that porting sigtool and codesign_allocate (both of which are open source) would make it pretty easy to add signing support to Go.

[kubkon]

@leo60228 oh for sure. I was just wondering if you knew if it was fixed; then porting codesign_allocate functionality would amount to finding the offset of string table, and padding out a fixed size slice at string table offset plus its size. But since the size of the signature can be variable, this has to be taken into account, and I guess that the staleness would indeed break because of it, unless the code signature command and the matching section are ignored from this calculation. Anyhow, we'll be figuring out something similar for Zig, so I'll be more than happy to share any exciting news of success (if any).

[cherrymui]

Thanks all above for thoughts and pointers! Here is my plan:

• Before stamping the buildid, allocate the LC_CODE_SIGNATURE load command with the right offset and size, zero-fill the signature blob. Probably done by the linker.
• Stamp the buildid. The buildid would be made to be insensitive to the LC_CODE_SIGNATURE blob. (but can be sensitive to the load command itself, which should not change)
• Fill in the signature blob. Since the buildid is insensitive to the signature blob, this shouldn't change the buildid.

I guess this may work. I'll try to code up something...

[kubkon]

That's my plan for Zig as well minus the buildid step ;-) I also plan to stretch the structure a little bit in the sense I really don't like the fact that codesign_allocate bullies me into having no gaps between the __LINKEDIT hidden sections. I wonder if this is indeed strictly required for the code signature to be properly generated.

[nikitavoloboev]

You don't have to pay anyone to code sign things right? It's just about running codesign <path-to-binary> and then sharing it with everyone?

[kubkon]

You don't have to pay anyone to code sign things right? It's just about running codesign <path-to-binary> and then sharing it with everyone?

As long as we're talking about adhoc code signing, then yep, that's the case. The full command for this would actually be codesign -s - <path_to_binary>.

[nikitavoloboev]

As long as we're talking about adhoc code signing

And then the users would be still met with a warning. Like this app is from unidentified developer, you sure you want to run it? or it will be okay to run as if everything is normal?

[rfay]

I should mention that real codesigning can be bundled up several ways without leaning the whole process. https://github.com/drud/signing_tools has the tools that we use (in Catalina) for signing our golang binaries. It does require an Apple developer account. And they do break it regularly by requiring you to sign off on contract changes.

[kubkon]

As long as we're talking about adhoc code signing

And then the users would be still met with a warning. Like this app is from unidentified developer, you sure you want to run it? or it will be okay to run as if everything is normal?

They will indeed. The GateKeeper is still there. If you want to please it, you gotta code sign with an actual, valid developer identity.

A little bit more about adhoc code signing, not sure if this was mentioned here or not, but @andrewrk and myself have exchanged a couple of emails with Apple officials, and they told us that modification of an already signed file in-place will end up in SIGKILL 9 by the kernel even if you regenerate the signature. That's because the kernel apparently caches code signing info for each inode. Interestingly, you probably won't notice this when using codesign tool for instance, as it actually does a full file copy under-the-hood. This probably doesn't hurt Go as much as it does Zig where we perform incremental linking in-place.

[terinjokes]

That's because the kernel apparently caches code signing info for each inode.

I don't have an M1 to test with, so this might be a dumb question. Does modifying a codesigned executable in place invalidate this cache?

[kubkon]

That's because the kernel apparently caches code signing info for each inode.

I don't have an M1 to test with, so this might be a dumb question. Does modifying a codesigned executable in place invalidate this cache?

Yep, it unfortunately does. Then, you either have to copy the entire contents into a new inode and code sign anew (thinking here of codesigning solutions that are part of Go/Zig's toolchains), or use codesign tool which will do that for you.

[networkimprov]

This is the Apple docs on running an app from an unidentified developer. In Catalina, it's considerably more complicated than previously -- you can't just Ctrl-click the app icon and select "Open" to get a dialog with a run-anyway option.

https://support.apple.com/en-us/HT202491

Has this changed in Big Sur?

[jpap]

A little bit more about adhoc code signing, not sure if this was mentioned here or not, but @andrewrk and myself have exchanged a couple of emails with Apple officials, and they told us that modification of an already signed file in-place will end up in SIGKILL 9 by the kernel even if you regenerate the signature. That's because the kernel apparently caches code signing info for each inode. Interestingly, you probably won't notice this when using codesign tool for instance, as it actually does a full file copy under-the-hood. This probably doesn't hurt Go as much as it does Zig where we perform incremental linking in-place.

My understanding is that the inode caching only gets in the way if you first execute the binary before (re-)codesign. Does Zig execute a binary before a final codesign step?

[andrewrk]

Yes, it's part of the edit-compile-run development cycle. Programmer edits code. Compiler updates the binary in place. Programmer runs & debugs the application. Repeat.

[kubkon]

Oh no, you're right. I guess I didn't explain myself well. That's the use case I had in mind. In Zig we're working on hot code swapping and incremental linking, so you can run the binary and then recompile just bits of it. Everything is done in place.

[jpap]

Oh no, you're right. I guess I didn't explain myself well. That's the use case I had in mind. In Zig we're working on hot code swapping and incremental linking, so you can run the binary and then recompile just bits of it. Everything is done in place.

Ah that all makes perfect sense. Thanks for the reply @andrewrk and @kubkon. Go performs builds in a new $TMPDIR each time the build cache is stale, so I guess each unique binary will have a different inode, even when moved into its final directory to replace an existing executable.

[andrewrk]

Yep we will be forced to do something similar with the new code signing requirements. Most likely performing the in-place update within the zig-cache directory, and then copying the binary to the output each time. Sadly Darwin has no copy_file_range syscall.