notarize MacOS desktop app

[iurimatias]

Symptoms

Lack of notarization results in a warning when running/installing the DMG file:

Details

According to Apple this failure is due to lack of something called "notarization":

Notarization gives users more confidence that the Developer ID-signed software you distribute has been checked by Apple for malicious components. Notarization is not App Review. The Apple notary service is an automated system that scans your software for malicious content, checks for code-signing issues, and returns the results to you quickly. If there are no issues, the notary service generates a ticket for you to staple to your software; the notary service also publishes that ticket online where Gatekeeper can find it.

https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution

[jakubgs]

Not sure if another issue was necessary: https://github.com/status-im/infra-ci/issues/24

[iurimatias]

it helps to keep track on our board

[jakubgs]

As mentioned in https://github.com/status-im/infra-ci/issues/24 XCode provides a CLI tool for uploading the app for notarization:

Unpublished Software. It’s easy to get unpublished software notarized with the Export process or xcodebuild. Custom build workflows are supported by the xcrun altool command line tool for uploading, and you can use xcrun stapler to attach the ticket to the package.

https://developer.apple.com/developer-id/

[jakubgs]

This article shows an example of usage of xcrun altool:

% xcrun altool --notarize-app
               --primary-bundle-id "com.example.ote.zip"
               --username "AC_USERNAME"
               --password "@keychain:AC_PASSWORD"
               --asc-provider <ProviderShortname>
               --file OvernightTextEditor_11.6.8.zip

https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/customizing_the_notarization_workflow https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution

[jakubgs]

And regarding 2FA:

Add the username and password options to supply your App Store Connect credentials. Because App Store Connect now requires two-factor authentication (2FA) on all accounts, you must create an app-specific password for altool, as described in Using app-specific passwords.

https://support.apple.com/en-us/HT204397

[jakubgs]

For future reference:

administrator@macos-03.ms-eu-dublin.ci.misc:~ % xcrun altool --help
Copyright (c) 2009-2020, Apple Inc. Version 4.029.1194

Usage: altool --validate-app -f <file> -t <platform> {-u <username> [-p <password>] | --apiKey <api_key> --apiIssuer <issuer_id>}
       altool --upload-app -f <file> -t <platform> {-u <username> [-p <password>] | --apiKey <api_key> --apiIssuer <issuer_id>}
       altool --notarize-app -f <file> --primary-bundle-id <bundle_id> {-u <username> [-p <password>] | --apiKey <api_key> --apiIssuer <issuer_id>} [--asc-provider <provider_shortname> | --team-id <wwdr_team_id>]
       altool --notarization-info <uuid> {-u <username> [-p <password>] | --apiKey <api_key> --apiIssuer <issuer_id>}
       altool --notarization-history <page> {-u <username> [-p <password>] | --apiKey <api_key> --apiIssuer <issuer_id>} [--asc-provider <provider_shortname> | --team-id <wwdr_team_id>]
       altool --list-apps {-u <username> [-p <password>] | --apiKey <api_key> --apiIssuer <issuer_id>}
       altool --list-providers {-u <username> [-p <password>] | --apiKey <api_key> --apiIssuer <issuer_id>}
       altool --store-password-in-keychain-item <name_for_keychain_item> -u <username> -p <password>

Exit codes: 0 success, 1 failure (Upon failure, an error code and message are generally also displayed.)

Authentication: Most commands require authorization.
                There are two methods available: user name with password, and apiKey with apiIssuer.

                -u, --username <username> Username. Required to connect for validation, upload, and notarization.
                -p, --password <password> Password. Required if username specified. If this argument is not supplied on the command line,
                                          it will be read from stdin.
                                          Alternatively to entering <password> in plaintext, it may also be specified using a '@keychain:'
                                          or '@env:' prefix followed by a keychain password item name or environment variable name.
                                          Example: '-p @keychain:<name>' uses the password stored in the keychain password item named <name>.
                                                                         You can create and update keychain items with the
                                                                         --store-password-in-keychain-item command. Note also that the
                                                                         --username can be inferred from the keychain item so --username
                                                                         can be omitted when using a '-p @keychain:' option.
                                          Example: '-p @env:<variable>'  uses the value in the environment variable named <variable>

                --apiKey <api_key>        apiKey. Required for JWT authentication while using validation, upload, and notarization.
                                          This option will search the following directories in sequence for a private key file
                                          with the name of 'AuthKey_<api_key>.p8':  './private_keys', '~/private_keys', '~/.private_keys',
                                          and '~/.appstoreconnect/private_keys'.
                --apiIssuer <issuer_id>   Issuer ID. Required if --apiKey is specified.

 -f, --file <file>                   <file> specifies the path to the file to process.
 -t, --type {osx | ios | appletvos}  Specify the platform of the file.

     --primary-bundle-id <bundle_id> Used with --notarize-app to uniquely identify a package.

     --asc-provider <provider_shortname> Required with --notarize-app and --notarization-history when a user account is associated with multiple
                                         providers. You can use the --list-providers command to retrieve the providers associated with
                                         your account. You may instead use --team-id and specify your WWDR Team ID.

     --team-id <wwdr_team_id>        Required  with --notarize-app and --notarization-history when a user account is associated with multiple
                                     providers. You can use the --list-providers command to retrieve the providers associated with
                                     your account. You may instead use --asc-provider and specify your App Store Connect short name.

 -v, --validate-app                  Validates an app archive for the App Store. Authentication and -f are required.
     --upload-app                    Uploads the given app archive to the App Store. Authentication and -f are required.
     --list-apps                     Display all apps associated with your account(s).

     --list-providers                Displays a list of the providers associated with your account along with short name and team id.
                                     This command is useful to determine what short name to use with the --asc-provider and --team-id options.

     --notarize-app                  Uploads the given app package, dmg or zip file for notarization. Authentication, -f,
                                     and --primary-bundle-id are required. --asc-provider or --team-id is required for an account associated with multiple providers.
                                     If successful, the UUID associated with the upload is returned.

     --notarization-info <uuid>      Returns the status and log file URL of a package previously uploaded for notarization with the specified <uuid>.
                                     Authentication is required. The log file can be retrieved with 'curl <log_file_url>'.

     --notarization-history <page>   Returns a list of all uploads submitted for notarization. <page> specifies a range of entries where 0
                                     returns the most recent number of entries. A new page value will be returned which can be used as the
                                     <page> value to the next use of --notarization-history and so forth until no more items are returned.
                                     Authentication is required. --asc-provider or --team-id is required for an account associated with multiple providers.

     --store-password-in-keychain-item <name_for_keychain_item> -u <username> -p <password>
                                     Stores the password <password> in the keychain item named <name_for_keychain_item> associated with the account <username>.
                                     If an item with that name and account already exists in the keychain, its password will be updated. Otherwise a new item
                                     is created with that name. You can use this keychain item with the -p option to mask your password with other commands.
                                     Example: altool --store-password-in-keychain-item MY_SECRET -u jappleseed@apple.com -p "MyP@ssw0rd!@78"
                                              altool --notarize-app -u jappleseed@apple.com -p @keychain:MY_SECRET [...]

     --transport <transport(s)>      Allows you to specify the protocol used when using --upload-app or --notarize-app. You should only use this option when
                                     instructed by Apple. <transport(s)> can be one or more of the following separated by commas: Signiant, Aspera, DAV, HTTPS.

     --output-format {xml | json | normal} Specifies how the output is formatted. 'xml' and 'json' display the output in a structured format; 'normal' displays in
                                           an unstructured format (default).

     --verbose                       Enable logging output.

 -h, --help                          Display this output.

[jakubgs]

To get list of ASC providers I need to log in:

administrator@macos-03.ms-eu-dublin.ci.misc:~ % xcrun altool --list-providers
2021-05-26 11:58:48.163 altool[7443:17178070] *** Error: Failed to retrieve providers info.
2021-05-26 11:58:48.163 altool[7443:17178070] *** Error: code -1011 (You must specify authentication credentials (username/password or apiKey/apiIssuer). Unable to list providers.)

[jakubgs]

According to this doc:

When you click Next, Xcode uploads your archive to the notary service. When the upload is complete, the notary service begins the scanning process, which usually takes less than an hour. While the notary service scans your software, you can continue to prepare your archive for distribution. For example, you can export the archive and perform any final testing that you require prior to making your software available to customers.

When the notarization process finishes, Xcode downloads the ticket and staples it to your archive. At that point, export your archive again to receive a distributable version of your software that includes the notary ticket.

https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution

Which means that finishing the notarization process as part of the same Jenkins build would take an hour or most probably multiple hours, which would be a problem for QA team or devs that would like to start testing and verifying the build as soon as possible.

[jakubgs]

It might make more sense to finish the notarization process using a separate job using:

Notarization produces a ticket that tells Gatekeeper that your app is notarized. After notarization completes successfully, the next time any user attempts to run your app on macOS 10.14 or later, Gatekeeper finds the ticket online. This includes users who downloaded your app before notarization.

You should also attach the ticket to your software using the stapler tool, so that future distributions include the ticket. This ensures that Gatekeeper can find the ticket even when a network connection isn’t available. To attach a ticket to your app, bundle, disk image, or flat installer package, use the stapler tool:

% xcrun stapler staple "Overnight TextEditor.app"

Though it should also be possible to distribute the app without "stapling" the ticket to the app.

[jakubgs]

Oh, apparently it might be faster than that:

Notarization completes for most software within 5 minutes, and for 98% of software within 15 minutes. However, notarization can take longer under certain conditions. Help to avoid long response times by following a few rules:

• Minimize the total number of files, even when the individual files are small.
• Don’t save non-executable files in places that require code signatures, like MyApp.app/Content/MacOS/. Instead, save these files to a directory that doesn’t require a code signature, like MyApp.app/Contents/Resources/.
• Don’t submit corrupted disk images. You can verify the integrity of a disk image by running the hdiutil utility: % hdiutil verify <image>
• Avoid heavily compressed disk images.
• Avoid huge, non-executable data files, especially when they change often.
• Limit notarizations to 75 per day.

https://developer.apple.com/documentation/security/notarizing_macos_software_before_distribution/customizing_the_notarization_workflow

So it is possible that if we do this right the notarization process can be quite fast.

The Limit notarizations to 75 per day. part suggests we should avoid notarizing PR builds at all.

[jakubgs]

This script could be used for inspiration: https://github.com/rednoah/notarize-app

[jakubgs]

I guess my question would be what value should be used for --primary-bundle-id? @iurimatias

I guess since mobile uses im.status.ethereum something like im.status.ethereum.desktop? Not sure if using the same value would work. Wish we used im.status.ethereum.mobile from the start...

[iurimatias]

im.status.ethereum.desktop SGTM

[jakubgs]

When using --output-format json with xcrun altool we get output like this:

{
  "tool-version": "4.029.1194",
  "tool-path": "/Applications/Xcode.app/Contents/SharedFrameworks/ContentDeliveryServices.framework/Versions/A/Frameworks/AppStoreService.framework",
  "notarization-upload": {
    "RequestUUID": "fa8c413d-dd16-44e9-83c5-59aa8abdf2c6"
  },
  "success-message": "No errors uploading pkg/StatusIm-Desktop-v0.1.0-beta.9-1786f2.dmg.",
  "os-version": "10.15.7"
}

Or:

{
  "tool-version": "4.029.1194",
  "tool-path": "/Applications/Xcode.app/Contents/SharedFrameworks/ContentDeliveryServices.framework/Versions/A/Frameworks/AppStoreService.framework",
  "success-message": "No errors getting notarization info.",
  "notarization-info": {
    "Hash": "ff5d568055c1109dbb19fbf93ed8c22c863766927aac3c24c9cca4d602076c67",
    "Status": "in progress",
    "RequestUUID": "13474d86-941b-41b8-838d-4a106eea5cc6",
    "Date": "2021-05-27T10:39:54.000Z"
  },
  "os-version": "10.15.7"
}

[jakubgs]

It also appears we get emails whenever a bundle is notarized:

[jakubgs]

And there we have it! A successful notarization:

{
  "tool-version": "4.029.1194",
  "tool-path": "/Applications/Xcode.app/Contents/SharedFrameworks/ContentDeliveryServices.framework/Versions/A/Frameworks/AppStoreService.framework",
  "success-message": "No errors getting notarization info.",
  "notarization-info": {
    "Status": "success",
    "Status Message": "Package Approved",
    "LogFileURL": "https://osxapps-ssl.itunes.apple.com/itunes-assets/Enigma125/v4/f0/d8/01/f0d80138-29cb-7d0e-d2bb-456441d46d9c/developer_log.json?accessKey=123qwe%123qwe%123qwe%123qwe%123qwe",
    "Date": "2021-05-27T10:39:54.000Z",
    "RequestUUID": "13474d86-941b-41b8-838d-4a106eea5cc6",
    "Status Code": 0,
    "Hash": "ff5d568055c1109dbb19fbf93ed8c22c863766927aac3c24c9cca4d602076c67"
  },
  "os-version": "10.15.7"
}

https://ci.status.im/job/status-desktop/job/platforms/job/ci-macos-notarization/12/console

[jakubgs]

Notarization of our app takes about 5-7 minutes:

12:02:16  + make notarize-macos
12:02:16  
12:02:16  ### Creating Notarization Request...
12:02:50  
12:02:50  ### Request ID: 91adbcb5-91c9-4017-805c-4eabc3db305d
12:02:50  
12:02:50  ### Checking Notarization Status...
12:03:19  In progress, sleeping 30s...
12:03:53  In progress, sleeping 30s...
12:04:21  In progress, sleeping 30s...
12:04:55  In progress, sleeping 30s...
12:05:29  In progress, sleeping 30s...
12:05:58  In progress, sleeping 30s...
12:06:32  In progress, sleeping 30s...
12:07:00  In progress, sleeping 30s...
12:07:34  In progress, sleeping 30s...
12:08:03  In progress, sleeping 30s...
12:08:37  In progress, sleeping 30s...
12:09:11  
12:09:11  ### Successful Notarization
12:09:11  
12:09:11  ### Stapling Notarization Ticket...
12:09:11  Processing: /Users/jenkins/workspace/status-desktop/platforms/ci-macos-notarization/pkg/StatusIm-Desktop-v0.1.0-beta.9-39582e.dmg
12:09:11  Processing: /Users/jenkins/workspace/status-desktop/platforms/ci-macos-notarization/pkg/StatusIm-Desktop-v0.1.0-beta.9-39582e.dmg
12:09:11  The staple and validate action worked!

https://ci.status.im/job/status-desktop/job/platforms/job/ci-macos-notarization/14/console

But I'm sur ewe could make that a bit faster if we follow some suggestions from https://github.com/status-im/status-desktop/issues/2169#issuecomment-848690577.

[iurimatias]

@michaelsbradleyjr could you please review https://github.com/status-im/status-desktop/pull/2600