macOS signs and notarizes without issue, Linux fails notarization.

[MrAlex94]

Hi @indygreg, great work you've done on this.

I seem to have hit an odd issue when building gecko-dev and trying to notarize on Linux. It may hold clues to issues experienced in #28 and #35.

The toolchain used for building Gecko on both macOS and Linux is LLVM 14 directly from Mozilla (XCode is NOT used on macOS).

rcodesign 0.22.0 is used on both platforms, and identical commands are run.

• Linux host, build targeting macOS, notarization fails.
• Linux host, build made on macOS and moved to Linux host, notarization fails.
• macOS host, build made on Linux and moved to macOS host, notarization fails.
• macOS host, build targeting macOS, notarization succeeds.

You would expect the macOS built .app that signs and notarizes okay on macOS to also sign and notarize without issue on Linux, but it doesn't.

You would also expect the Linux built .app to notarize without issue on macOS, but it doesn't.

The script for signing and notarizing:

#!/bin/bash
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at https://mozilla.org/MPL/2.0/.
#

set -x

usage ()
{
  echo  "Usage: $0 "
  echo  "    -a <PATH-TO-BROWSER.app>"
  echo  "    -i <IDENTITY>"
  echo  "    -s <SECRET>"
  echo  "    -b <ENTITLEMENTS-FILE>"
  echo  "    -p <CHILD-ENTITLEMENTS-FILE>"
  exit 255
}

while getopts "a:i:s:b:p:" opt; do
  case ${opt} in
    a ) BUNDLE=$OPTARG ;;
    i ) IDENTITY=$OPTARG ;;
    s ) SECRET=$OPTARG ;;
    b ) BROWSER_ENTITLEMENTS_FILE=$OPTARG ;;
    p ) PLUGINCONTAINER_ENTITLEMENTS_FILE=$OPTARG ;;
    \? ) usage; exit 255 ;;
  esac
done

rcodesign sign --p12-file "$IDENTITY" --p12-password "$SECRET" --code-signature-flags runtime --team-name <RETRACTED> \
"${BUNDLE}/Contents/MacOS/XUL"

find "${BUNDLE}"/Contents/MacOS -name '*.dylib' -exec \
rcodesign sign --p12-file "$IDENTITY" --p12-password "$SECRET" --code-signature-flags runtime --team-name <RETRACTED> {} \;

rcodesign sign --p12-file "$IDENTITY" --p12-password "$SECRET" --code-signature-flags runtime --team-name <RETRACTED> \
"${BUNDLE}"/Contents/MacOS/updater.app

rcodesign sign --p12-file "$IDENTITY" --p12-password "$SECRET" --code-signature-flags runtime --team-name <RETRACTED> \
-e "${BROWSER_ENTITLEMENTS_FILE}" \
"${BUNDLE}"/Contents/MacOS/waterfox-bin

rcodesign sign --p12-file "$IDENTITY" --p12-password "$SECRET" --code-signature-flags runtime --team-name <RETRACTED> \
-e "${BROWSER_ENTITLEMENTS_FILE}" \
"${BUNDLE}"/Contents/MacOS/waterfox

rcodesign sign --p12-file "$IDENTITY" --p12-password "$SECRET" --code-signature-flags runtime --team-name <RETRACTED> \
-e "${BROWSER_ENTITLEMENTS_FILE}" \
"${BUNDLE}"/Contents/Library/LaunchServices/net.waterfox.updater

find "${BUNDLE}"/Contents/Resources/gmp-clearkey -type f -exec \
rcodesign sign --p12-file "$IDENTITY" --p12-password "$SECRET" --code-signature-flags runtime --team-name <RETRACTED> {} \;

rcodesign sign --p12-file "$IDENTITY" --p12-password "$SECRET" --code-signature-flags runtime --team-name <RETRACTED> \
-e "${BROWSER_ENTITLEMENTS_FILE}" "${BUNDLE}"

rcodesign sign --p12-file "$IDENTITY" --p12-password "$SECRET" --code-signature-flags runtime --team-name <RETRACTED> \
-e "${PLUGINCONTAINER_ENTITLEMENTS_FILE}" --exclude '**' \
"${BUNDLE}"/Contents/MacOS/plugin-container.app

# Notarise and staple the main bundle
rcodesign notary-submit \
  --api-key-path ./key.json \
  --staple \
  "${BUNDLE}"

Linux host, Linux built (FAIL).

notary log> {
notary log>   "archiveFilename": "Waterfox.app.zip",
notary log>   "issues": [
notary log>     {
notary log>       "architecture": "x86_64",
notary log>       "code": null,
notary log>       "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087735",
notary log>       "message": "The signature of the binary is invalid.",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/waterfox",
notary log>       "severity": "error"
notary log>     },
notary log>     {
notary log>       "architecture": "x86_64",
notary log>       "code": null,
notary log>       "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087735",
notary log>       "message": "The signature of the binary is invalid.",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/updater.app/Contents/MacOS/net.waterfox.updater",
notary log>       "severity": "error"
notary log>     }
notary log>   ],
notary log>   "jobId": "9f311b7a-e5af-42f9-812c-27863042da6f",
notary log>   "logFormatVersion": 1,
notary log>   "sha256": "b83799b4a80573a7b3f47962f0b08f625500d64bbbf683492cd3bb60eb8be91b",
notary log>   "status": "Invalid",
notary log>   "statusCode": 4000,
notary log>   "statusSummary": "Archive contains critical validation errors",
notary log>   "ticketContents": null,
notary log>   "uploadDate": "2023-01-17T10:27:12.480Z"
notary log> }
Error: notarization invalid

Linux host, macOS built (FAIL).

notary log> {
notary log>   "archiveFilename": "Waterfox.app.zip",
notary log>   "issues": [
notary log>     {
notary log>       "architecture": "x86_64",
notary log>       "code": null,
notary log>       "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087735",
notary log>       "message": "The signature of the binary is invalid.",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/waterfox",
notary log>       "severity": "error"
notary log>     },
notary log>     {
notary log>       "architecture": "x86_64",
notary log>       "code": null,
notary log>       "docUrl": "https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/resolving_common_notarization_issues#3087735",
notary log>       "message": "The signature of the binary is invalid.",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/updater.app/Contents/MacOS/com.waterfox.updater",
notary log>       "severity": "error"
notary log>     }
notary log>   ],
notary log>   "jobId": "c431a7ed-cf6c-4903-b183-d116f042f13f",
notary log>   "logFormatVersion": 1,
notary log>   "sha256": "bd7b87c2f5264dcd793847fbd5b07a731471068e31ecc2790575435b17a65115",
notary log>   "status": "Invalid",
notary log>   "statusCode": 4000,
notary log>   "statusSummary": "Archive contains critical validation errors",
notary log>   "ticketContents": null,
notary log>   "uploadDate": "2023-01-17T11:15:08.459Z"
notary log> }
Error: notarization invalid

macOS host, macOS built (SUCCESS):

notary log> {
notary log>   "archiveFilename": "Waterfox.app.zip",
notary log>   "issues": null,
notary log>   "jobId": "4a24235f-3cf4-4f10-a153-042b8bbe21f7",
notary log>   "logFormatVersion": 1,
notary log>   "sha256": "5191159260e1a240faf0fb786885b3f42fed32ea6bf30a601af5c881e7fbca0f",
notary log>   "status": "Accepted",
notary log>   "statusCode": 0,
notary log>   "statusSummary": "Ready for distribution",
notary log>   "ticketContents": [
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "5afefd05c4d7461f04b49cc4c572ce20dee7ca25",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libmozavutil.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "44ee61204c8a21f84a6f9b8a0a9277a13a62acf1",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "aa1a48fd9c0dc7540e27d35100d43914a48b6f5e",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libsoftokn3.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "010791805858be49bd2588e4b460c30357b622f9",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libipcclientcerts.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "48bfb73b9eeb61dce88982953018551c72dafbe4",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libfreebl3.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "4b33f87453c0a41aef399fc0301cb98598a9421f",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libosclientcerts.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "231e6da005898ebe147c6a2d96652af2722c951e",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libnss3.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "d0025599b475ce55bfe7cb9f2abc2f9f2cbcc6a2",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/waterfox-bin"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "c4d9bd9033a50c40a2a8054465e63c7c064447fd",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/liblgpllibs.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "2a12cfb718d4173f5179f379c920e7e82c656d70",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/XUL"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "eec1036b7d585b11a73e12fc94303cb27dc74939",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/plugin-container.app"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "4082ed5343a9b4d36f8492b06a8ad7980b18cf8a",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libmozglue.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "73bd10aa9f424f559e4ff45f857e8f6be9816e98",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libmozavcodec.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "a10a22e78278272f2b6ec1fece40dbf43ff2ba39",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libnssckbi.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "48bfb73b9eeb61dce88982953018551c72dafbe4",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libfreebl3.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "c4d9bd9033a50c40a2a8054465e63c7c064447fd",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/liblgpllibs.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "aa1a48fd9c0dc7540e27d35100d43914a48b6f5e",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libsoftokn3.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "2a12cfb718d4173f5179f379c920e7e82c656d70",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/XUL"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "4b33f87453c0a41aef399fc0301cb98598a9421f",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libosclientcerts.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "5afefd05c4d7461f04b49cc4c572ce20dee7ca25",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libmozavutil.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "4082ed5343a9b4d36f8492b06a8ad7980b18cf8a",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libmozglue.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "010791805858be49bd2588e4b460c30357b622f9",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libipcclientcerts.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "73bd10aa9f424f559e4ff45f857e8f6be9816e98",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libmozavcodec.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "44ee61204c8a21f84a6f9b8a0a9277a13a62acf1",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/waterfox"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "d0025599b475ce55bfe7cb9f2abc2f9f2cbcc6a2",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/waterfox-bin"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "a10a22e78278272f2b6ec1fece40dbf43ff2ba39",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libnssckbi.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "231e6da005898ebe147c6a2d96652af2722c951e",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/libnss3.dylib"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "eec1036b7d585b11a73e12fc94303cb27dc74939",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/plugin-container.app"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "eec1036b7d585b11a73e12fc94303cb27dc74939",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/MacOS/plugin-container.app/Contents/MacOS/plugin-container"
notary log>     },
notary log>     {
notary log>       "arch": "arm64",
notary log>       "cdhash": "5c949e72d5befb9e3b2af0a084aff0eaafb89a82",
notary log>       "digestAlgorithm": "SHA-256",
notary log>       "path": "Waterfox.app.zip/Waterfox.app/Contents/Resources/gmp-clearkey/0.1/libclearkey.dylib"
notary log>     }
notary log>   ],
notary log>   "uploadDate": "2023-01-17T11:38:40.018Z"
notary log> }

[indygreg]

Symptoms seem to be pointing to a difference between the build performed on macOS vs Linux. I suspect that codesign -v will catch issues as well - so you could remove notarization from your debugging loop to make things faster.

We'll need to compare the Linux and macOS builds - both before and after [r]codesign signing - to isolate where they disagree. Once we do that, we can find the source of the bug in rcodesign.

The rcodesign diff-signatures command should be useful.

[indygreg]

We should also try signing just the updater.app bundle and just the updater.app/Contents/MacOS/com.waterfox.updater binary inside to attempt to isolate if the failure is:

1. Signing an individual Mach-O binary.
2. Signing a bundle.
3. Signing a nested bundle.

Apple's validation code sometimes attributes the source of an error to a distant component, not the actual component with the error. So you need to sign things independently to attempt to isolate the real failure.

[MrAlex94]

Okay! So I've made a build on macOS, and signed one version with codesign and the other with rcodesign. I couldn't use rcodesign diff-signatures, because it was spitting out Error: YAML serialization error: serializing nested enums in YAML is not supported yet.

Attached are:

• codesign.bash - The script to sign with codesign.
• codesign.log - The output of running codesign.bash
• codesign.yml - The output of codesign print-signature-info Waterfox-codesign.app
• rcodesign.bash - The script to sign with rcodesign
• rcodesign-1.log - The output of running rcodesign.bash, just signing the plugin-container.app bundle. (No updater in this build).
• rcodesign-2.log - The output of running rcodesign.bash, just signing the plugin-container binary.
• rcodesign-1.yml - The output of rcodesign print-signature-info Waterfox-rcodesign.app, with the bundle signed.
• rcodesign-2.yml - The output of rcodesign print-signature-info Waterfox-rcodesign.app, with the binary signed.

codesign.bash.txt rcodesign.bash.txt

codesign.log codesign.yml.txt

rcodesign-1.log rcodesign-1.yml.txt rcodesign-2.log rcodesign-2.yml.txt

[MrAlex94]

@indygreg - just checking, is the above okay or is there any more info I can provide for you?

[gedw99]

Dumb question .. to notorise I still need to buy a 99 euro developer very from Apple ?

[roblabla]

Dumb question .. to notorise I still need to buy a 99 euro developer very from Apple ?

Yes, notarization requires an Apple Developer membership.

[gedw99]

thanks @roblabla things have not changed for the better :)

[indygreg]

I suspect - but haven't verified - that this issue was due to a bug in how we recursively traverse nested bundles. I just pushed a bunch of changes to the main branch in the past few hours to hopefully fix all the remaining issues. So I'm going to close this issue.

Please reopen if there are still problems with signing Waterfox.