Signing the spin executable breaks it

[mikkelhegn]

Spin version 2.5.1

Please see this thread dor context: https://discord.com/channels/926888690310053918/1250561699254108201

Tl;dr: Rancher signed the spin executable as part of distributing it wiht Rancher Desktop. Howver signing the binary breaks the spin up functionality as soon as a client connects to the app.

[jandubois]

Sample crash report:

spin.crash-report.txt

Removing the signature "fixes" the spin up functionality:

codesign --remove-signature "/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/bin/spin"

[lann]

Some relevant lines from the crash report:

Process:               spin [41590]
Parent Process:        spin [41588]
OS Version:            macOS 12.7.5 (21H1222)
Exception Type:        EXC_BAD_ACCESS (SIGKILL (Code Signature Invalid))

Spin re-executes itself as part of spin up's trigger executor framework (note the Parent Process: spin above). I suspect that this self-re-execution is somehow interacting badly with macOS's code signing enforcement but I don't use a Mac so someone else may need to investigate further.

[lann]

Oh this is also interesting:

The crash report gives the binary path as /Users/USER/*/spin; I assume USER/* is just crash report anonymization, but notably this is under /Users/.

@jandubois's "remove codesigning" snippet gives the path as /Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/bin/spin.

If these reflect the actual paths of the parent (spin up) and child (trigger executor) processes, then it may be that spin up is actually executing a different spin binary rather than itself. It makes sense to me that a signed binary might be prohibited from calling an unsigned binary.

Edit: Reading through the code it isn't clear to me how this confusion would happen but it's still my best guess.

[jandubois]

It makes sense to me that a signed binary might be prohibited from calling an unsigned binary.

While spin invoking itself via a PATH search is indeed problematic (see also "spin build launches spin via a PATH search" in Adventures in bundling spin cli, I don't think it is the issue here:

$ PATH=/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
$ RD=/Applications/Rancher\ Desktop.app/Contents/Resources/resources/darwin/bin

$ # There is no other `spin` executable on the `PATH`
$ spin --version
-bash: spin: command not found
$ "$RD/spin" --version
spin 2.5.1 (cba6773 2024-05-14)

$ # `spin build` cannot lauch `spin` when it isn't on the `PATH`
$ "$RD/spin" build
Building component my-app with `npm run build`
…
webpack 5.91.0 compiled successfully in 91 ms
sh: spin: command not found
Error: Build command for component my-app failed with status Exited(127)

$ # When on the `PATH` spin can invoke itself from `spin build`
$ PATH="$RD:$PATH" spin build
Building component my-app with `npm run build`
…
Finished building all Spin components

So spin build can invoke the signed executable, but spin up fails (not immediately, but when a connection to localhost:3000 is made.

[lann]

not immediately, but when a connection to localhost:3000 is made.

Interesting. I'm not familiar with macOS code signing; is there some capability bit that needs to be turned on to allow network access or something?

[jandubois]

I'm not done with verification, but I've confirmed with local testing:

• In order to notarize the executable it needs to be configured to use the hardened runtime. I've confirmed that requesting the hardened runtime while codesigning is sufficient to trigger the broken spin up behaviour.

• I've confirmed that signing the binary without the hardened runtime does not break the spin up functionality.

• I've further confirmed that when signing the binary with the hardened runtime, but adding the com.apple.security.cs.allow-unsigned-executable-memory entitlement also allows spin up to work normally.

• I tried to use the com.apple.security.cs.allow-jit entitlement, but it alone did not work, so I assume that Spin is not using the "passing MAP_JIT flag to the mmap system function" mechanism to allocate its executable memory area.

I've added a special signing rule for spin to the Rancher Desktop signing config in https://github.com/rancher-sandbox/rancher-desktop/pull/7051 as a potential fix candidate.

[lann]

That all makes sense to me. Wasmtime as a runtime definitely requires execution of unsigned code and it doesn't do JIT compilation in any sense that would matter for mmap; I think of it more as "AOT right before executing".

[jandubois]

it doesn't do JIT compilation in any sense that would matter for mmap

I think it is just a particular method to manage executable code that is somewhat safer than allowing arbitrary unsigned executable memory access.

Basically allocating it with mmap with the MAP_JIT flag and then manipulating it with pthread_jit_write_protect_np. But I don't think we need to worry about it right now; requesting the broader entitlement seems fine for our current use cases. It might be different when you are targetting iOS and want to distribute your code via Apple's app store. 😄

[jandubois]

I'm not done with verification, but I've confirmed with local testing:

Fix has been confirmed now.

We will make a Rancher Desktop 1.14.2 release next week that has a resigned spin binary on macOS but no other changes.

Since this does not affect most users, we will not put it into the auto-update configuration; but just announce it on Slack and Discord so users know they can upgrade manually. But new users will get the fixed version automatically.