MacOS captive bundle signing: Cannot resolve "embedded framework contains modified or invalid version" error

Problem Description

I've spent 40+ hours in the past week trying to get a MacOS captive runtime bundle code signed and notarized. I've followed dozens of guides/threads found here, on the AIR dev site, in multiple Apple forums and elsewhere and always get the same end result:

The signature of the binary is invalid.

And when I verify my signatures, the result is always:

codesign -vvv --deep --strict /release-mac/MyApp.app

--prepared:/release-mac/MyApp.app/Contents/Frameworks/Adobe AIR.framework
--validated:/release-mac/MyApp.app/Contents/Frameworks/Adobe AIR.framework
/release-mac/MyApp.app: embedded framework contains modified or invalid version
In subcomponent: /release-mac/MyApp.app/Contents/Frameworks/Adobe AIR.framework

AND

codesign -vvv --strict /release-mac/MyApp.app/Contents/MacOS/MyApp

--prepared:/release-mac/MyApp.app/Contents/Frameworks/Adobe AIR.framework
--validated:/release-mac/MyApp.app/Contents/Frameworks/Adobe AIR.framework
/release-mac/MyApp.app/Contents/MacOS/MyApp: embedded framework contains modified or invalid version
In subcomponent: /release-mac/MyApp.app/Contents/Frameworks/Adobe AIR.framework

I've finally pulled the last hair out of my head, so I'm now asking for help. I've used Windows for all development work for the past 25 years, so I'm somewhat new to using a Mac for development. Hopefully, I'm just missing something simple.

Some Background

I'm converting a pretty large Flex application to AIR for a client so that they can stop using a hacked up portable version of Firefox embedded with an old version of Flash Player (it's used internally only). Based on Harman's recommendations, I am building captive runtime bundles with custom installers. I am not using any ANEs. It's a straight-up Flex app that used to run in a browser that connects to a backend ColdFusion server.

Once I got my code signing certificate for Windows, I was able to produce a Windows installer using InnoSetup. I had never used InnoSetup before, and it still only took about an hour to get a working signed installer. Super simple and easy!

Now I'm trying to create a pkg installer for their Mac users. I recently purchased a Mac Mini (M1) so that I could build MacOS captive runtime bundles and then create pkg installers. I have my Developer ID application/installer and Apple intermediate/root certificates installed and they seem to be working just fine. But like what most developers seem to face, the signing and notarization process changes often and what worked a couple months ago no longer applies. There is quite a bit of conflicting information too.

I use IntelliJ IDEA and can compile and run both debug and release versions of the apps on the Mac. Last week I was working with AIR SDK 50.1.1.1 and this week I tried 50.1.1.2 hoping the MacOS related packaging updates would help (they did not). Also, the Mac Mini is on my local network and I use Remote Desktop Manager to connect to it from my Windows system via Apple Remote Desktop (ARD), which is so much better and responsive compared to all the VNC solutions out there.

Things I've Tried

First, I use IntelliJ IDEA to build the release version of the app. I end up with a swf and the app descriptor xml file in the output folder. I then use a command line to package the bundle based on how IntelliJ runs the ADT command, replacing the pfx file signing parameters for the KeychainStore method of signing. It builds the app bundle and I can open it successfully.

java -Dapplication.home="$path_sdk" -Dfile.encoding=UTF-8 -Djava.awt.headless=true -Duser.language=en -Duser.region=en -Xmx2048m -jar "$path_adt" \
  -package -storetype KeychainStore -alias "$cert_app" -target bundle "$path_app" "$path_xml" -C "$path_out" "$file_swf" -C "$path_src" "$file_ico"

Note: to see the IntelliJ IDEA command being used, you can use an invalid application namespace in the descriptor file (http://ns.adobe.com/air/application/50.1BAD for example), then Build > Package AIR Application... as a captive runtime bundle. It will eventually throw an error due to the invalid namespace, and in the notifications panel you will see the error notification along with a link to the "ADT Command Line".

So when it comes to signing the app, I first tried following the guide on the AIR dev site, and that didn't work. However, based on other things I've read, signing the main app bundle first and then signing the framework and main executable doesn't make sense, so I tried signing the framework, main executable, and then the app bundle. Same issues as above.

I have also tried the ADT package command where you repeat the storetype/alias parameters after the -target bundle, to have ADT sign everything for you with entitlements, and again, same issues as above. I have also tried resigning the Adobe Air dynlib file buried within in the framework folder as a first step, same result. I think I've tried every conceivable combination of signing the dynlib file, framework, main executable and/or app bundle in every possible order with no success. I also tried not using the "runtime option" parameter on the the framework per some Apple docs.

I ran ADT using both the JRE/JBR that IntelliJ IDEA comes bundled with (a custom version of Java 17) and also a fresh install of Java 11. Both seemed to work the same.

What else...

I am using the 4 entitlements as shown on the AIR dev site. I have not added any other entitlements or Mac specific settings to the app descriptor file. When testing the entitlements, they are confirmed to be there.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>com.apple.security.cs.allow-dyld-environment-variables</key>
    <true/>
    <key>com.apple.security.cs.allow-jit</key>
    <true/>
    <key>com.apple.security.cs.allow-unsigned-executable-memory</key>
    <true/>
    <key>com.apple.security.cs.disable-library-validation</key>
    <true/>
</dict>
</plist>

*** Checking Main Executable Entitlements ***

Executable=/release-mac/MyApp.app/Contents/MacOS/MyApp
[Dict]
    [Key] com.apple.security.cs.allow-dyld-environment-variables
    [Value]
        [Bool] true
    [Key] com.apple.security.cs.allow-jit
    [Value]
        [Bool] true
    [Key] com.apple.security.cs.allow-unsigned-executable-memory
    [Value]
        [Bool] true
    [Key] com.apple.security.cs.disable-library-validation
    [Value]
        [Bool] true

When I "codesign -dvvv" any signed file or bundle, those always appear to be as expected (authorities are: Developer ID Application, Developer ID Certification Authority, and Apple Root CA).

When I codesign and then immediately verify each step, the framework seems OK (see below), and then after signing the main executable afterwards is when the "embedded framework contains modified or invalid version" issue appears, before signing the entire bundle. Results of signing the framework before signing the main executable:

*** Signing Adobe Air Framework ***

/release-mac/MyApp.app/Contents/Frameworks/Adobe Air.framework: signed bundle with generic [Adobe AIR]

*** Testing Adobe Air Framework Signature ***

/release-mac/MyApp.app/Contents/Frameworks/Adobe Air.framework: valid on disk
/release-mac/MyApp.app/Contents/Frameworks/Adobe Air.framework: satisfies its Designated Requirement

I tried a utility called ArchiChect to help diagnose the issue. One interesting find is when I drag the runtimes/air-captive/mac/Adobe Air.framework/Versions/1.0 folder to it from the SDK folder (since it has a _CodeSignatures folder in it), it shows:

spctl error 3
/Users/jroberts/Developer/SDKs/Flex/4.16.1-50.1.1-32/runtimes/air-captive/mac/Adobe AIR.framework/Versions/1.0: rejected
source=Unnotarized Developer ID
origin=Developer ID Application: Harman Connected Services Inc (ZUQHAK4PRS)

If I drag over the 1.0 folder from the packaged bundle, it shows the same thing. If I resign the Adobe Air file in it, then it has the same rejected source with my Developer ID Application certificate name as the origin.

At one point, I recall using otool or something to list linked libraries, but couldn't make any sense out of it. I wish the "embedded framework contains modified or invalid version" error had more information about what specific embedded framework has the issue, where its embedded, and which specific version number it's complaining about. I don't know if it's something within the main executable or something in the AIR framework.

I recall the otool output having references to other frameworks scattered all over. Are there other SDKs that I need to install?

Once I can resolve this "embedded framework contains modified or invalid version" issue, I'm assuming that packaging the app and getting it notarized won't result in "The signature of the binary is invalid." errors any longer.

I can't think of any other useful details. Hopefully someone out there knows what I might be missing or could be doing wrong. The process seems simple enough once you understand it, so maybe it's not a problem with me, but with the AIR SDK?

UPDATE: I think I figured out what the version issue is about.

When I codesign "Adobe Air.framework", the signature info includes:

CodeDirectory v=20200 size=201 flags=0x0(none) hashes=1+3 location=embedded

While the main executable (MacOS/MyApp) and app bundle are showing:

CodeDirectory v=20500 size=912 flags=0x10000(runtime) hashes=18+7 location=embedded

I believe it's the v=20200 that is the "invalid version", and I'm not sure why it's not signing with v=20500 like everything else.

The framework signature also shows:

Format=bundle with generic

and

Info.plist=not bound

I'm assuming the format is also incorrect. I'm not sure if Info.plist being not bound is incorrect or not.

Upon further investigation, I believe the symlinks within the "Adobe Air.framework" folder are not working correctly (and in my SDK folder). It could be how I extracted the files from AIRSDK_Flex_MacOS.zip. It would make sense that bad symlinks would cause all of these problems, because it can't actually find the inner "Adobe Air" file and "Resources" folder. But why would the app work when I run it then?

In the morning, I'm going to rebuild my SDK folder a different way, rebuild the project and bundle and see if that helps.

I recall some other folk had been talking about symlink issues that were causing signing problems... so in case this helps, what I've got here in a signed app bundle is:

In the framework folder, Versions is a folder which contains another folder called 1.0
In the Versions folder is a symlink to 1.0 which is called Current
At that top-level framework, as well as Versions there's also a symlink called Adobe AIR -> Versions/Current/Adobe AIR
Plus alongside that there's a final symlink called Resources -> Versions/Current/Resources
In the Versions/1.0 folder, we do have a symlink called Adobe AIR_64 which just points to Adobe AIR. I'd asked about this one, it's there for some historic requirement from when Apple first started supporting 64-bit binaries, so it might be something that can be deleted now.. (and then of course everything would need to be code-signed again).
Also in Versions/1.0 is the Resources folder and the generated _CodeSignature folder

In terms of the code directory version, afaik that's just something you can determine when you do the code signing. If you've got a recent Mac Mini then I'm assuming it's also updated with Ventura or similar so I'd assume it's an up-to-date version of the codesign tool, but the flags show that your command is missing the --options runtime part? Not sure whether that alone is enough to trigger the update to the later version.. the command we use is essentially what's shown in that linked article:

codesign -f --options runtime -s "Developer ID Application: etc" "bundle.app/Contents/Frameworks/AdobeAIR.framework"

(actually I think we always include entitlements in our command lines, even through I believe they're ignored for frameworks)

I have also tried the ADT package command where you repeat the storetype/alias parameters after the -target bundle, to have ADT sign everything for you with entitlements, and again, same issues as above.

This is the step that I really want to make sure works... so when we find out what the issue is, I'm hoping we can work backwards and update ADT so that it's ensuring that the code signatures are fully valid...

Happy to try to analyse the signed bundle if you want to send it to us?

thanks

Hi Andrew,

There was no way I was going to sleep when I felt I was so close to solving this puzzle. I rebuilt my Flex/AIR SDK, rebuilt the project, ran my ADT bundle script and now everything works! I even got the final pkg file notarized and stapled! I confirmed that my symlinks were definitely not extracted properly when I did the original SDK build.

I still need to test the installer on a different system somehow though. Is creating a new user account good enough or does it need to be a system without any of the developer certificates installed on it or some other criteria?

Here is what the "Adobe Air.framework" signature now shows:

Format=bundle with Mach-O universal (x86_64 arm64)
CodeDirectory v=20400 size=126929 flags=0x0(none) hashes=3960+3 location=embedded
Info.plist entries=21

I first signed "Adobe Air.framework": codesign -s "$cert_app" -f --timestamp -v --strict "$path_air"

NOTE: I chose to exclude -o runtime and entitlements per Apple examples, because it's a code bundle and not a main executable. The internal binary is the main executable and already signed. I could be misunderstanding their rules though. When I do add the runtime option and entitlements, the signature then shows v=20500 instead of v=20400, along with the runtime flag.

I did not resign the "Adobe Air" binary in the framework folder. However, upon further investigation, it appears that signing the framework bundle actually resigns the binary inside (it's now signed by me and not Harman). And if you don't specify entitlements when signing the framework bundle, the original 6 entitlements that the Harman signed binary had disappear. So I will update my script to add the runtime option and entitlements when signing the framework bundle. codesign -s "$cert_app" -f --timestamp -o runtime --entitlements "$path_ent" -v --strict "$path_air"

I then signed the main executable: codesign -s "$cert_app" -f --timestamp -o runtime --entitlements "$path_ent" -v --strict "$path_exe"

I then signed the bundle: codesign -s "$cert_app" -f --timestamp -o runtime --entitlements "$path_ent" -v --strict "$path_app"

I also tested the packaging method where ADT signs everything for you. It also came back notarized and I was able to staple it.

When I compare the two app folders (individually signed with runtime+entitlements on all 3 codesigns vs ADT automated), the file sizes of the Adobe Air binary and main app executable are slightly larger in the "ADT automated" version, each 80 bytes larger. The signatures are identical except for the hashes in both sets of files, and the "Signature Size" is different in the main executable signature where the "individual" version is 9045 and the "automated" version is 9046. All the entitlements are the same with both sets of files. I'm not sure why there are differences, but I noticed it and thought I would report it.

Is there any way to avoid having to retype my login password a dozen times when signing everything?

Thanks for the feedback!

Awesome well done (and I hope you can sleep well now!)

So does that mean that it was the fact there were file copies, rather than symlinks, that was behind the actual issue?

I chose to exclude -o runtime and entitlements per Apple examples, because it's a code bundle and not a main executable. The internal binary is the main executable and already signed. I could be misunderstanding their rules though.

I think you're right here but we had initially been using the same scripts to sign the framework bundle as the main application bundle, and it worked so we didn't change it. I also think the rules are not the clearest (neither are the error messages!)

signing the framework bundle actually resigns the binary inside

Yes -> I'm also wondering whether this is part of the issue, if they look for and find a binary frameworkname.framework/frameworkname then perhaps this is what's signed but then the code signature iterates through and finds another binary under frameworkname.framework/Versions/Current/frameworkname and then the two don't match...

Also if you sign an app bundle, it modifies the executable too, in the same way. So e.g. codesign appname.app is the same as codesign appname.app/Contents/MacOS/appname

the file sizes of the Adobe Air binary and main app executable are slightly larger in the "ADT automated" version

There could be a couple of differences here including the formatting of the DER-encoded signature, or just some padding .. a lot of the time these aspects don't make a difference (but you never know ...!)

Is there any way to avoid having to retype my login password a dozen times when signing everything

Not sure - there must be I think, because our build scripts don't need passwords to be typed in, but they do need to be run in a proper terminal window (even if remote-accessed) rather than via an ssh login. I recall trying to find a way to do this a long time ago, I think I found the "trust" setting on lots of the certificates and changed them .. and then codesigning just failed all the time with some weird message until I reset the trust settings on all the certificates (or maybe just the Apple ones)? It's a bit of a dark art I think....

I still need to test the installer on a different system somehow though. Is creating a new user account good enough or does it need to be a system without any of the developer certificates installed on it or some other criteria?

If your certificates are in the Keychain app under the "login" item, then yes it's fine to create a new user account, they wouldn't then be available to the other user. If they're in "system" then it may be worth checking on another machine.

Quick thought -> if this was caused by the extracted SDK having file-copies instead of sym-links in that Adobe AIR.framework location, then this is actually something we should able to at least detect, if not correct, when we package up the bundle....

thanks

So does that mean that it was the fact there were file copies, rather than symlinks, that was behind the actual issue?

Yes. I could open the symlink files and they contained text of the symlink path, such as Versions/Current/Adobe Air instead of opening the target file or directory. Also, ls -l did not show the symlink info such as Adobe Air -> Versions/Current/Adobe Air.

Quick thought -> if this was caused by the extracted SDK having file-copies instead of sym-links in that Adobe AIR.framework location, then this is actually something we should able to at least detect, if not correct, when we package up the bundle....

Yes, that is what happened. If you can detect bad symlinks in the SDK, and either fix the SDK symlinks or recreate the symlinks in the generated bundle, that would help others that break them like I did. I can help test that if/when you add that.

Also if you sign an app bundle, it modifies the executable too, in the same way.

So that means out of the 4 "code items" in my app bundle -- the framework binary, the framework bundle, the main app binary and the main app bundle -- I should really only need to sign the 2 bundles since the related binaries also get signed in the process. I just tested that and everything looks as expected. All 4 code items have my signatures and all the entitlements, and it was notarized and stapled without issues.

If your certificates are in the Keychain app under the "login" item, then yes it's fine to create a new user account, they wouldn't then be available to the other user. If they're in "system" then it may be worth checking on another machine.

I did install the certificates to login and System per the first walk-through that I tried from the AIR dev site. Maybe I can remove them from System since I believe the signing process is using the login certificates anyways, hence all the passwords I need to enter. I'll search around for a way to eliminate having to enter password to codesign. I recall seeing something related to during my endless searching trying to resolve my issues.

If there is anything else you want me test, or look into, let me know! I appreciate your help.

Jason

airsdk / Adobe-Runtime-Support