Support MacOS ad hoc signing on all platforms

JeyJeyGao commented 1 year ago

Description

I am currently working on a cross-platform binary project, notation-azure-kv, based on .NET. We have recently run into an issue where our cross-compiled binary is unable to function properly on macOS due to the absence of macOS CodeSign.

As you might know, the CodeSign process can only be executed on macOS. This makes the release procedure for cross-platform binaries more complicated, as every developer is required to construct their own macOS-specific pipeline in order to integrate the CodeSign.

Given this, I suggest that .NET should implement macOS CodeSign in its offerings instead of call the codesign binary on macOS, so the codesign can be done on any OS. By doing so, it would greatly simplify the CodeSign process and would make .NET a more appealing and efficient language for cross-platform binary products.

Moreover, if this functionality was combined with .NET's Ahead-of-Time (AOT) compilation feature, .NET would be even more robust and efficient as a Command Line Interface (CLI) language. This would streamline the development process for all .NET developers and potentially open up .NET to more use-cases in the future.

Reproduction Steps

On an Linux machine, run:

dotnet publish \
    --configuration Release \
    --self-contained true \
    -p:PublishSingleFile=true \
    -r osx-arm64 \
    -o "$output_dir/binary-name"

to build an macOS binary. Then run the binary on ARM based macOS machine, the binary will be killed because of no codesign.

Expected behavior

The codesign should be done in a platform independent way like Golang. https://github.com/golang/go/blob/master/src/cmd/internal/codesign/codesign.go

Actual behavior

If the macOS binary was not built on macOS, the binary cannot run on an ARM macOS machine. The user needs to do codesign manually.

Regression?

No response

Known Workarounds

No response

Configuration

No response

Other information

No response

am11 commented 1 year ago

HostWriter invokes codesign tool on macOS machine https://github.com/dotnet/runtime/blob/7486bdead70b884598c3288f1dfc232b2f077cc1/src/installer/managed/Microsoft.NET.HostModel/AppHost/HostWriter.cs#L149 if we implement the managed codesign tool like in your go example, this restriction will go away.

ghost commented 1 year ago

Tagging subscribers to this area: @vitek-karas, @agocke See info in area-owners.md if you want to be subscribed.

Issue Details

### Description I am currently working on a cross-platform binary project, [notation-azure-kv](https://github.com/Azure/notation-azure-kv), based on .NET. We have recently run into an issue where our cross-compiled binary is unable to function properly on macOS due to the absence of macOS CodeSign. As you might know, the CodeSign process can only be executed on macOS. This makes the release procedure for cross-platform binaries more complicated, as every developer is required to construct their own macOS-specific pipeline in order to integrate the CodeSign. Given this, **I suggest that .NET should implement macOS CodeSign in its offerings instead of call the `codesign` binary on macOS**, so the codesign can be done on any OS. By doing so, it would greatly simplify the CodeSign process and would make .NET a more appealing and efficient language for cross-platform binary products. Moreover, if this functionality was combined with .NET's Ahead-of-Time (AOT) compilation feature, .NET would be even more robust and efficient as a Command Line Interface (CLI) language. This would streamline the development process for all .NET developers and potentially open up .NET to more use-cases in the future. ### Reproduction Steps On an Linux machine, run: ```sh dotnet publish \ --configuration Release \ --self-contained true \ -p:PublishSingleFile=true \ -r osx-arm64 \ -o "$output_dir/binary-name" ``` to build an macOS binary. Then run the binary on ARM based macOS machine, the binary will be `killed` because of no codesign. ### Expected behavior The codesign should be done in a platform independent way like Golang. https://github.com/golang/go/blob/master/src/cmd/internal/codesign/codesign.go ### Actual behavior If the macOS binary was not built on macOS, the binary cannot run on an ARM macOS machine. The user needs to do codesign manually. ### Regression? _No response_ ### Known Workarounds _No response_ ### Configuration _No response_ ### Other information _No response_

Author:	JeyJeyGao
Assignees:	-
Labels:	`area-HostModel`, `untriaged`, `needs-area-label`
Milestone:	-

filipnavara commented 1 year ago

This has been discussed time and again. codesign is only one of the tools you need to produce the app bundles. You also need the resource compilers, cross-compiler for native code for any non-trivial use case, and the SDK (which is only licensed to be used on Mac hardware). In fact, codesign is the easiest one to make cross-platform.

agocke commented 1 year ago

Mac has two requirements:

On ARM64, every binary must be signed or it will be killed at startup. An ad hoc signature is fine to avoid this problem.
Gatekeeper requires a signed binary with a valid developer ID

If a codesign substitute is simple enough to integrate, I'd consider adding it to solve the first problem. We will not attempt to solve the second problem. If you want to release an official binary for Mac, you need to do it with a Mac.

agocke commented 1 year ago

Note: because of the above I would want to only integrate exactly the code necessary to ad-hoc sign a binary, no more.

filipnavara commented 1 year ago

If a codesign substitute is simple enough to integrate, I'd consider adding it to solve the first problem.

That would not work. The ad-hoc code signing can only be done on the Mac where the executable is run. Not only it cannot be done on Windows, it cannot be done on any other machine at all. The whole point is that the signature is registered with the kernel for the particular version of file based on its hash.

agocke commented 1 year ago

I thought this rule was changed in an update. I'll try to find the cite.

agocke commented 1 year ago

I can't find the cite, but I tried this using two Mac machines: one x64 and one ARM64.

I built a self-contained .NET app targeting osx-arm64 on the x64 machine, then copied it to the ARM64 machine and ran it. It seems to work. In contrast, when I disable codesigning, or build on a non-Mac machine, the binary is killed immediately on launch. The codesign headers appear to corroborate that the Mac binary is ad-hoc signed and that this is satisfactory. The Mac binary:

$ codesign -dv singlefile 
Executable=/Users/angocke/tmp/singlefile
Identifier=singlefile-5555494499eb084541733bab894e4531f837a4be
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=95756 flags=0x2(adhoc) hashes=2986+2 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

The Linux binary:

$ codesign -dv sf8 
sf8: code object is not signed at all

am11 commented 1 year ago

The Linux binary:

$ codesign -dv sf8 
sf8: code object is not signed at all

after that:

# see:
# * intro: https://gregoryszorc.com/blog/2022/08/08/achieving-a-completely-open-source-implementation-of-apple-code-signing-and-notarization/
# * repo: https://github.com/indygreg/apple-platform-rs/tree/main/apple-codesign#rcodesign-cli
# * PR branch: https://github.com/indygreg/apple-platform-rs/pull/78
$ cargo install --git https://github.com/melvyn2/apple-platform-rs --branch pr --bin rcodesign apple-codesign

$ rcodesign --help

Sign and notarize Apple programs. See https://gregoryszorc.com/docs/apple-codesign/main/ for more docs.

Usage: rcodesign [OPTIONS] [COMMAND]

Commands:
  analyze-certificate                   Analyze an X.509 certificate for Apple code signing properties
  compute-code-hashes                   Compute code hashes for a binary
  diff-signatures                       Print a diff between the signature content of two paths
  encode-app-store-connect-api-key      Encode App Store Connect API Key metadata to a single file
  extract                               Extracts code signature data from a Mach-O binary
  generate-certificate-signing-request  Generates a certificate signing request that can be sent to Apple and exchanged for a signing certificate
  generate-self-signed-certificate      Generate a self-signed certificate for code signing
  keychain-export-certificate-chain     Export Apple CA certificates from the macOS Keychain
  keychain-print-certificates           Print information about certificates in the macOS keychain
  notary-log                            Fetch the notarization log for a previous submission
  notary-submit                         Upload an asset to Apple for notarization and possibly staple it
  notary-wait                           Wait for completion of a previous submission
  parse-code-signing-requirement        Parse binary Code Signing Requirement data into a human readable string
  print-signature-info                  Print signature information for a filesystem path
  smartcard-scan                        Show information about available smartcard (SC) devices
  smartcard-generate-key                Generate a new private key on a smartcard
  smartcard-import                      Import a code signing certificate and key into a smartcard
  remote-sign                           Create signatures initiated from a remote signing operation
  sign                                  Sign a Mach-O binary or bundle
  staple                                Staples a notarization ticket to an entity
  verify                                Verifies code signature data
  x509-oids                             Print information about X.509 OIDs related to Apple code signing
  help                                  Print this message or the help of the given subcommand(s)

Options:
  -v, --verbose...  Increase logging verbosity. Can be specified multiple times.
  -h, --help        Print help
  -V, --version     Print version

$ rcodesign sign publish/exe1

then copy publish/ dir to macOS (I'm on arm64) and app works:

$ docker cp 2a00ed66ea4366c78f7883dfdd74707a582da2f01dafe6bf67c1cf3374b917d9:/exe1/publish/ publish
$ codesign -dv publish/exe1

Executable=/Users/adeel/projects/exe1/publish/exe1
Identifier=exe1
Format=Mach-O thin (arm64)
CodeDirectory v=20400 size=734909 flags=0x2(adhoc) hashes=22961+2 location=embedded
Signature=adhoc
Info.plist=not bound
TeamIdentifier=not set
Sealed Resources=none
Internal requirements count=0 size=12

JeyJeyGao commented 1 year ago

@am11 The apple-platform-rs project is impressive, and it seems to function well. We prioritize a toolchain that is trustworthy, reliable, and offers long-term support. If .NET could provide ad-hoc code-signing capabilities similar to Golang, it would greatly benefit many developers.

agocke commented 1 year ago

@JeyJeyGao As mentioned, any solution in this space would be unsuitable for releasing binaries to customers, due to Apple restrictions. Does that still provide any benefit for you?

alexrp commented 1 year ago

It seems like ad-hoc signing would still be useful when releasing developer-oriented tooling?

agocke commented 1 year ago

Perhaps? Depends on how strict Gatekeeper will be in the future + how much you care about forcing workarounds on users.

alexrp commented 1 year ago

Also, fwiw, Zig also supports this: https://github.com/ziglang/zig/issues/7103

Perhaps? Depends on how strict Gatekeeper will be in the future + how much you care about forcing workarounds on users.

To be frank: If I'm releasing open source utility programs for use by a small niche community, I'm not going to go out of my way to pay for a certificate and signing infrastructure just to deal with Apple's nonsense. I'm ok with telling those niche users to either disable Gatekeeper for that program or globally.

JeyJeyGao commented 1 year ago

I agree that for proprietary software, developers should have a certificate issued by Apple and sign the binary on a macOS machine. However, within the open-source community, there are many developers who start with small projects, and an ad-hoc code sign is sufficient to make it work on macOS.

Large-scale adoption of the GitHub release tool for cross-platform .NET binary products is still lacking. In comparison, for Go, we have tools like GoReleaser, which produce output that can be directly executed on all platforms. For .NET, we have to construct the release pipeline ourselves and require a macOS machine for code signing, which can be expensive.

I believe the current code signing capabilities for .NET are good, but they require more effort from the user compared to other languages.

agocke commented 1 year ago

I'm convinced that ad-hoc signing is worth supporting. I'd be willing to integrate it directly into the SDK, if someone wants to go through the effort of porting the code to do ad hoc signing to C#.

filipnavara commented 1 year ago

if someone wants to go through the effort of porting the code to do ad hoc signing to C#.

https://github.com/filipnavara/CodeSign

agocke commented 1 year ago

@filipnavara Looks like it's MIT, mind if we integrate it? 😄

filipnavara commented 1 year ago

I made it MIT specifically because I was hoping that some of the Microsoft teams adapt it instead of rolling their own versions.

Orgl9l commented 10 months ago

Apologies of this comment is not supposed be be here I have no idea when well to look or go:

Given that GitHub has a MacOS build chain with MacOS runners (that have Xcode tools installed), and specific instruction for Installing an Apple certificate on macOS runners for Xcode development it would be great to understand what else in missing - "for want of a nail".

There seem to be no .Net/MSFT documentation on this (that I have been able to easily find - so find myself here and other issues https://github.com/dotnet/sdk/issues/34514, https://github.com/dotnet/runtime/pull/53913 and am not sure of this is the correct pace to post; but figure that given:

It seems "today" that locally built/published .Net executable single file runs on my M1 Mac fine, that:
- Same should occur on the GitHub runner on basis that the above linked GitHub instruction are followed.

However of note, the clear omission from the above linked instructions states that:

Note: For iOS build targets, your provisioning profile should have the extension .mobileprovision. For macOS build targets, the extension should be .provisionprofile. The example workflow above should be updated to reflect your target platform.

while one can (perhaps incorrectly) interpolate this change:

PP_PATH=$RUNNER_TEMP/build_pp.provisionprofile

vs:

PP_PATH=$RUNNER_TEMP/build_pp.mobileprovision

it does not cover what changes need to be here:

# apply provisioning profile
          mkdir -p ~/Library/MobileDevice/Provisioning\ Profiles
          cp $PP_PATH ~/Library/MobileDevice/Provisioning\ Profiles

Does that dotnet publish signing only work on an M1 runner and not the Mac Intel runner...?

One also hope/assume that any solution here will also work for AoT builds too!

Any guidance from the community would be hugely appreciated!

also @alexrp this is no cost to get a developer cert from Apple

dotnet / sdk