j9ac9k commented 4 years ago

I have been working on notarizing a fbs generated .app bundle. I know fbs currently does not support code signing and notarization, but I would be happy to post what I've learned; and various steps and fixes along the way if it would help integrate these features at some point. I currently have my application codesigned, and am working on the notarization process.

I'm not sure where the appropriate place to put that information would be; hence opening the issue and seeing what you would prefer.

Thanks again for making this awesome package.

mherrmann commented 4 years ago

Thank you for your offer! That would be very interesting indeed. Please just post the info here; with your permission i'll then integrate it into fbs, or the docs.

j9ac9k commented 4 years ago

I apologize I don't have this in guide form; I'll write this out and embed links as relevant. I realized I wasn't explicit in my original post but feel free to take the content I post here and reword/rephrase/embed into docs as you see fit. if you find the information useful for integrating into fbs, by all means, please do so!

I should also point the lessons/pitfalls I've come across are by no means exhaustive, I can only comment on issues I have come across, or issues I've seen reports of.

Motivation

Starting with macOS 10.15, the default security settings will be to require all applications be notarized.

Notarization involves sending your .dmg file (which has your .app bundle embedded) to Apple, where they verify a number of things, they return a certificate that you "staple" to your .app bundle prior to deploying. Notarization requires the following:

Code signing for all distributed executables
Application was signed with a valid Developer ID certificate
Application has hardened runtime enabled (this is done during codesigning step)
Secure timestamp is part of the code signing signature
Linked against the macOS 10.9 or later SDK

Before you get started

You will need the following things:

XCode 10 or later
Developer ID Application Certificate in your keychain
Apple Developer Login credentials to the role of Developer, App Manager, Account Holder or Admin

You can get this certificate by signing up for an Apple Developer account, or joining an existing group and having the group admin give you access to the key. This is required for codesigning. For notarization, you will need your own apple developer account, with a application specific password (to get around the 2FA); it is recommended that these credentials are stored in your non-iCloud keychain.

Prepare the Bundle

fbs currently uses pyinstaller and generates the .app bundle. The output bundle needs further modification to conform to codesigning requirements. Specifically, codesign treats any directory in ./Contents/MacOS/ that have a period in the name as separate bundles. This is a problem when using QML, as there are many directories such as QtQuick.2 and so on. BotoTig (github user) was kind enough to create a script and post it on the PyInstaller Wiki. While this covers most cases, It did not address issues with "translations" and "support" folders, so I made a modification to the script to move those folders from ./Contents/MacOS/ to ./Contents/Resources/. My version of the script can be seen here.

Code Signing

After fixing the bundle, you are now ready to codesign the package. Most apple documentation on the issue is in context to using XCode, so it's a bit different for Python/Qt based packages. The command I use is:

codesign -fs "$CODESIGN_ID" --deep --force --verbose -o runtime --preserve-metadata=identifier,entitlements,requirements,runtime --timestamp ./target/MyPkg.app

Where $CODESIGN_ID is the name of the developer ID certificate in my keychain (in my case it's "Developer ID Application: SomeCompany, Inc. (XX##XXXX##)" (where X is a letter and # is a number).

Some of the documentation is explicit for not using --deep, however for our purposes it is necessary as we are not building the project inside XCode.

The next step we want to do is simulate "moving" the .app bundle. The reason we do this is part of the code signing signature checks the extended attributes, so if we verified the codesign signature, it may return "Valid", but the moment the .app bundle is moved, the codesign verficiation could fail. To simulate the move run the command

xattr -cr ./target/MyPkg.app

Next step is to verify the codesign is valid, I do that through two separate methods:

codesign --verify --verbose=4 ./target/MyPkg.app
spctl --assess --verbose ./target/MyPkg.app

This should both return no errors, or "valid" results. When this is done, you now have a .app bundle that is codesigned.

Notarization

This link will be of reference

First thing you will want to do is have your apple developer ID credentials in the keychain, along with your application specific password. The link above has the steps to do that.

The next step is to notarize the application. As we cannot upload a directory, we will upload the .dmg file that fbs install creates for us.

xcrun altool --notarize-app --primary-bundle-id "com.company.group.application" --username "<apple  developer email>" --password "@keychain:AC_PASSWORD" --file ./target/MyPkg.dmg

You will get a bundle ID to check the status of it later, along with an email if it is rejected or accepted.

Caveat At The Time of this Writing

Currently PySide2 5.12.4 and 5.13.0 (haven't tested other versions) have libpyside2.abi3.5.12.dylib and libshiboken2.abi3.5.12.dylib linked against macOS SDK 10.0 (at least the version from PyPI), meaning they cannot be notarized (notarization requires SDK 10.9 or newer).

You can download/build PySide2 locally to get around this (I have not done this yet). I have opened a bugreport here. If this is relevant to you, I would encourage commenting there.

I typed this out in a bit of a rush, no doubt I missed stuff, @mherrmann let me know where you would like more detail/references and I'll edit this post to reflect that.

mherrmann commented 4 years ago

Thank you very much for this. I need to get access to macOS 10.13+ so I can have XCode 10, which you wrote is required. I'll post back here.

j9ac9k commented 4 years ago

I should update that I got my application to notarize, but it was not easy!

Compile custom pyside2 wheels myself (and then use them when building my application):

python setup.py bdist_wheel --qmake=/Users/gitlab-runner/Fred/Qt5.12.4/5.12.4/clang_64/bin/qmake --build-tests --ignore-git --parallel=8 --standalone --macos-sysroot=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk

Modify the codesign command to enable secure timestamps and hardened runtime

codesign -fs "$CODESIGN_ID" --deep --force --verbose -o runtime --entitlements .entitlements --preserve-metadata=identifier,entitlements,requirements,runtime --timestamp ./target/Fred.app

NumPy 1.17.0 was not working on building the package, but 1.16.4 does (I have not investigated this, but i suspect this is a pyinstaller issue)
I needed to remove any kind of writing to disk for my log file (currently working on solving this issue)

Here is the contents of my .entitlements file:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.cs.allow-jit</key>
    <true/>
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
    <true/>
    <key>com.apple.security.cs.allow-dyld-environment-variables</key>
    <true/>
    <key>com.apple.security.cs.disable-library-validation</key>
    <true/>
    <key>com.apple.security.cs.disable-executable-page-protection</key>
    <true/>
    <key>com.apple.security.files.user-selected.read-write</key>
    <true/>
</dict>
</plist>

j9ac9k commented 4 years ago

Lastly, I'm going to write up a detailed blog posts about my experience, and various edge cases I ran into. I do hope that my experience can help improve fbs, so please don't hesitate to ask any questions.

I do want to say the notarization process is easy to do by hand, but any kind of automated fashion via build pipeline will be challenging due to the asynchronous nature of the process (upload package to Apple, wait on approval, you can check the status via command line tool, ..you're given an email when accepted/rejected (status check tool will say pending, failed or accepted as well)).

mherrmann commented 4 years ago

Just an update on this: I'm afraid I'm extremely busy right now and will not get to this soon, also because it seems it's not required for me to do this for fman. I have an old Apple Developer Certificate, which (I think) does not require notarization. If my situation changes, I will update here.

j9ac9k commented 4 years ago

Notarization will be required with macOS 10.15 which is to be released pretty soon.

Not sure how this would be integrated into fbs, I'm envisioning a reference in documentation for the time being.

rickrcomm commented 2 years ago

My fbs "release"d PyQt5 app fulfill.app only runs from the /Applications .app bundle if I double-click the fulfill Unix executable inside the bundle. It works fine from there. But if I start fulfill by clicking on the bundle fulfill.app in /Applications, the icon shows up for a second or two in the dock and then disappears. I can't find any error message. Looked in console and have Sentry configured and tested for the app. It is NOT codesigned. Shouldn't I get an error popup from MacOS if that is the problem?

Disregard, I set full disk access in System Permissions and it solved the problem. Sorry to bother.

mherrmann / fbs-tutorial

Codesigning and Notarizing fbs built .app bundles #29

Motivation

Before you get started

Prepare the Bundle

Code Signing

Notarization

Caveat At The Time of this Writing