search

airsdk / Adobe-Runtime-Support

Report, track and discuss issues in Adobe AIR. Monitored by Adobe - and HARMAN - and maintained by the AIR community.
203 stars 11 forks source link

iOS Dynamic Libraries not signed correctly #521

Closed marchbold closed 3 years ago

marchbold commented 3 years ago

Problem Description

This has been an issue for a long time, I raised it with Adobe years ago but have just worked around it since then.

There are still certain 3rd party dynamic frameworks that AIR fails to sign correctly, examples include:

The response I got from Adobe at the time was

My observation is that, concerned dynamic framework OneSignal.framework was already signed by OneSignal. With the current implementation, AIR won’t consider them (already signed frameworks) for signing during packaging. AIR would rather expect dynamic frameworks to be unsigned before packaging.

They seemed to resolve it for a single framework in AIR 30, however a second framework of similar nature would still cause it to break.

Additionally there are certain circumstances which will cause a framework to fail, eg using FBSDKCore.framework on it's own works, but if you add certain native extensions with packaged frameworks it will fail, eg, combine with com.distriqt.GoogleIdentity, each work on their own, but as soon as they are combined the FBSDKCore.framework is no longer signed.

I feel this is related somehow to only signing one of the already signed dynamic frameworks.

Steps to Reproduce

Add the FBAudienceNetwork.framework to your Frameworks folder and attempt to package your iOS app.

You will get an error like:

ERROR: Install failed. Got error "ApplicationVerificationFailed" with code 0xe8008017: 
   Failed to verify code signature of /var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.9pdwzi/extracted/Payload/DynamicFrameworksTest.app/Frameworks/FBAudienceNetwork.framework :

Similarly add FBSDKCore.framework and com.distriqt.GoogleIdentity you will get a similar error 

ERROR: Install failed. Got error "ApplicationVerificationFailed" with code 0xe8008001: 
   Failed to verify code signature of /var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.rpApth/extracted/Payload/DynamicFrameworksTest.app/Frameworks/FBSDKCoreKit.framework : 0xe8008001 (An unknown error has occurred.)

Known Workarounds

Workaround is to resign the application manually ensuring the frameworks are signed using the correct profiles etc

https://github.com/distriqt/ANE-Adverts-Mediation/blob/master/lib/facebookaudience/scripts/resign

Resigning it this way and the application installs and everything works correctly.

arcreature7 commented 3 years ago

I'm building ipa in windows environment so this issue is very critical. Does Harman have any plans to fix this issue? I hope to receive a positive answer.

cgascons commented 3 years ago

We would also like this to be implemented as soon as possible, it's critical for our development pipeline.

urthling commented 3 years ago

Still seeing this issue on 33.1.1.345 -- I see from the release notes it was not included.. Any chance this can make it into the next build?

mrfrasier commented 3 years ago

+1 for addressing this issue. AIR is cross-platform, but a bug like this, forcing us to switch to a mac for a step in the process kills that. If we need a mac to compile, there are, frankly, much more robust solutions on the market than AIR

marchbold commented 3 years ago

@mrfrasier This isn't just a Windows issue, it affects macOS as well.

mrfrasier commented 3 years ago

@mrfrasier This isn't just a Windows issue, it affects macOS as well.

Correct, but the only known solution/workaround is Mac-Exclusive, which means we either need to use a Mac to proceed, or use something else.

urthling commented 3 years ago

Granted, this is a pain for sure -- note, you're not actually compiling on the MAC, you're just re-signing the IPA.. the only upside is that the process of signing is pretty straight forward so you don't need to switch platforms to continue development.

That said, I've commented out the FB ANE in order to continue development without having to sign the IPA every time.. And yes, Harman, really needs to find a solution for this for sure, fingers crossed, that will be sooner rather than later..

mrfrasier commented 3 years ago

So I have a solution for the Windows users. The app 3uTools now has an IPA signing tool. If you feed it your p12 and mobile provision it will sign the app and it should then install with no issue. An extra step for sure, but still better than having to switch to a mac to sign after every build.

Thankfully, 3u can also be used to install the app onto the device, so at least it does double duty.

urthling commented 3 years ago

Just tried the newly released .406 -- I was hopeful because the release notes mention signing -- but unfortunately, still not working. Running the code signing script is still necessary to allow the App to intall..

marchbold commented 3 years ago

@ajwfrost Any chance you can look into this issue while you are working on signing issues?

ajwfrost commented 3 years ago

@marchbold @urthling can I just check whether you've tried with our latest patched ADT.jar file? this should fix the signing of dylib files as well...

We'd been looking to get another fix done prior to our next SDK release but it's sounding like it would be useful to push this out earlier, so hopefully address this and the other signing-related problems..

urthling commented 3 years ago

@ajwfrost I'm running .406.. When you say patched, since .406, there's another ADT.jar per chance?

ajwfrost commented 3 years ago

Hi - yes, sorry, it was under #511 but that link is about to expire so please try with the below.. https://transfer.harman.com/message/EOmfgAn5M3SPSACcUPuG4I

urthling commented 3 years ago

Shucks, just tried, unfortunately still getting:

ERROR: Install failed. Got error "ApplicationVerificationFailed" with code 0xe8008016: Failed to verify code signature of ...

ajwfrost commented 3 years ago

@urthling If you're using MacOS (or that 3uTools thing mentioned above) are you able to re-sign the generated IPA file, and then have it install? if so, could you please send us both IPA files so that we can inspect the signatures and see what's wrong with it? There's precious little documentation on this (well, zero other than some comments in Apple's source code) so we may be missing something...

Link for uploading in case you need it: https://transfer.harman.com/requests/ffglXyutL8hlKCRF1oIpMr

thanks

urthling commented 3 years ago

You bet! Uploading now.. This is dev build and its resigning I've just uploaded, do you need a production build?

ajwfrost commented 3 years ago

Thanks - no that should be fine, we'll take a look to see what the differences are. The obvious one is a new file that's injected by the re-signing process, it seems very weird that we've never seen that before in anyone else's resigned packages - perhaps it's due to the use of this Windows tool. Curiously, it seems like this may be fairly key: https://stackoverflow.com/questions/21844728/what-is-the-archived-expanded-entitlements-xcent-file-in-ios-app But, there are also difference in the code signing settings embedded within each of the signed frameworks, so we'll look there first...

urthling commented 3 years ago

Sounds good! -- quick correction though, I'm on MAC.. I do see this toward the end of my signing script:

cp "$ENTITLEMENTS" "$WORKING_DIR/Payload/$APP_NAME/archived-expanded-entitlements.xcent"

marchbold commented 3 years ago

@urthling I think that line in the script was just one of the things we tried to resolve the issue and can actually be removed. So I would suggest ignoring the archived-expanded-entitlements.xcent file.

Here's our 2 example ipa's:

https://www.dropbox.com/s/d71atqfoifm9gom/dynamic-frameworks-example_ipa_files.zip?dl=0

Let me know if there's anything else you need to isolate this one. This is another issue that's been around since AIR first started signing dynamic frameworks...

ajwfrost commented 3 years ago

@urthling looking at your two files, they're signed with different certificates. The one that's signed by ADT is signed with "iPhone Distribution" with a company name vs the re-signed one that has "iPhone Developer" with a personal name.

Looking specifically at one of the framework files:

The obvious difference is that the re-signed one doesn't include the embedded entitlements - whereas we are putting these into the signature. But the entitlements there are basically empty.. and having looked into our signing code there's a comment from Adobe saying that frameworks are forbidden to have any entitlements, so perhaps that explains this difference.

There are also differences in the code directory signature block, most of these would be expected but the info.plist hash appears to be wrong in our version. This may actually be the only problem -- everything else relies on other things so much. There's an Info.plist file that sits alongside the binary and for some reason our code signature has the wrong values here for the hash which implies we're not correctly identifying this file whilst we're parsing the package (or we're accidentally recognising another fie as if it's an Info.plist file and are hashing that as well..)

I'll get a build of ADT done with a lot of extra debug output so that we can see what's happening. If you're able then to use the same certificate for both ADT and the re-signing, it just helps us compare things as the bytes align better (and the CMS blob should then be the same...)

thanks

urthling commented 3 years ago

@ajwfrost Ah, yes, my bad, since I'm now running the script after every build I forgot to update the actual cert since the last production build..

Ok, just re-ran it -- getting the same error, but happy to send this one on to you after using the updated ADT, using the same cert as one used in signing script. Per Michael's suggestion, I've also removed the line that mentions archived-expanded-entitlements.xcent

If you send me a link, I'll send on the ipa..

marchbold commented 3 years ago

That hash issue sounds hopeful! It was always confusing to me as to why it would sometimes sign correctly individually but not when multiples were added.

Also the ipa's in the link above should be with the same developer cert / profile.

ajwfrost commented 3 years ago

Hi @urthling @marchbold

Thanks for the details... a review of the code didn't show anything wrong, so we have created a version of ADT that should output some debug (actually, too much, having just tried it here, so you'll likely need to redirect the output into a file...). It hopefully will help us understand what's going on with the signing here, how we end up with the wrong hash for Info.plist.

Oddly I was expecting to see things being called twice, once for the armv7 binary and once for the arm64 one, but it looks like everything comes into our "signer" once, and then internally we put the code signatures into each of these components separately..

Anyway @urthling if you could please build using this ADT and do a redirect to a file and then send us the file - and the initial/resigned IPA pair that both are set up using the same signature, if that's possible too..

thanks

Link to download ADT: https://transfer.harman.com/message/7scocavQ569p0sMLFohfkF Link for file uploads (ignore the message): https://transfer.harman.com/requests/auohH26AdM5UfDsKLB7P8B

marchbold commented 3 years ago

@ajwfrost Just sent one through, (I've attempted an install at the end and you'll see the signing verification failure)

ajwfrost commented 3 years ago

Thanks @marchbold - so it's fairly clearly going wrong! It's just adding more and more data into the message that then gets hashed, so the first framework might be correct but subsequent ones would end up with a hash that's made up from the combinations of the previous frameworks' info.plist files...

The curious thing though is that for any one framework, we're seeing multiple Info.plist files that are the same size..

IPA output stream -> processing 754 for Frameworks/FBSDKShareKit.framework/FBSDKShareKit.framework/Info.plist
1: updating FrameworkInfoPlistDigest with 754 bytes
IPA output stream -> processing 754 for Frameworks/FBSDKShareKit.framework/Info.plist
1: updating FrameworkInfoPlistDigest with 754 bytes
Setting distribution provisioning profile -> false
1: setting framework entitlements
1: setting framework identifier: com.facebook.sdk.FBSDKShareKit
IPA output stream -> processing 1862 for Frameworks/FBSDKShareKit.framework/_CodeSignature/CodeResources
1: preparing to sign framework with FrameworkInfoPlistDigest

(the '1' at the start is just an instance identifier, I had hoped we would get a new one of these objects created for every file...)

Anyway, I think we can (a) ignore anything from a subfolder i.e. we'd ignore Frameworks/XXX.framework/XXX.framework/Info.plist and just hash Frameworks/XXX.framework/Info.plist itself), and (b) reset the hash every time we use it, so that we don't have this cumulative effect...

Will send an updated version out later..

thanks

ajwfrost commented 3 years ago

Hi again Are you able to try using this please? https://transfer.harman.com/message/Q9NOz9DXeqe2Bj6mGdbeiX thanks

urthling commented 3 years ago

@ajwfrost - this is a little odd, I'm using the updated ADT.jar - CPU usage spikes for several seconds, and then just sits @ 0.2%..? Reverting back to the 506 or earlier version - build then succeeds as normal..

marchbold commented 3 years ago

Output with that new version: output-2.txt

ajwfrost commented 3 years ago

Thanks @marchbold - I don't suppose the resulting IPA file worked..? Could I get this - and the re-signed one - to examine please? https://transfer.harman.com/requests/bY9i3Tmyn4xq2I09fGE064

@urthling not sure what was going wrong there, are you running this via an IDE or just from the command line? are you able to double-check the download worked, the output of shasum -a 256 adt.jar should be d2eff7fe986f14c83868d738e71eae68c12339dd013c6d5d90e12063ead8836a

thanks

marchbold commented 3 years ago

Ummmm sorry I think I broke that last build. It looks like it's working!?

(I had a framework in the Frameworks folder that wasn't related to any of the ANEs, note this seem to do something weird with copying the framework into itself which may explain the double paths you saw).

I've sent you the ipa and output log. I'll do some more tests with it today.

marchbold commented 3 years ago

I've just been doing some testing on Windows and this version is failing there for me with the following error: 

ld: unknown option: -platform_version

I don't get this with the original adt.jar.

ajwfrost commented 3 years ago

@marchbold that was caused by a change that should only have been in place for iOS-simulator, hence shouldn't have been set on Windows where we are using an older 'ld64' binary.. so, hopefully this is fixed now: https://transfer.harman.com/message/4SPAINrOXZzGEDmaAEUqNF

thanks

urthling commented 3 years ago

@ajwfrost uploading to you now using latest ADT.jar -- it's going to take a little while, connection is very slow from here..

urthling commented 3 years ago

AH! Just realized, the latest version works and installs onto my device! Well done, and thank you!!!!

ajwfrost commented 3 years ago

Great - thanks both for your help with getting us the samples to track down that problem. I just kicked off the release process for the SDK so next week we'll get this all out there formally...

cheers

ventr1x commented 3 years ago

I just tested it with 406 and the patched adt. It is not working here. I still have to use the resign script provided by distriqt or I get hit with "can't check integrity of the App" when trying to install on device.

ajwfrost commented 3 years ago

@ventr1x oh that's a pain! could you please upload the IPA file that was generated by ADT that doesn't work, plus the resigned version that does, using the below link? we can take a look at the diffs and see what else may need to change..

https://transfer.harman.com/requests/BYZ5TQJjDJ6uzrLJfJTsfr

thanks

marchbold commented 3 years ago

@ventr1x What frameworks are you using? I'd like to replicate too, I've only run a few tests so far but has been working on the couple I've tried.

ventr1x commented 3 years ago

Sorry that I ignored you, I am busy with quite a few different projects and jump around a lot.

We have really weird issues with the latest two AIR versions. An app around one year live suddenly cannot be installed by some of our testers with the app integrity error (mostly newest iOS and newer devices). I can install it just fine (iPhone XS Max iOS 14.4.x).

A new app in development, using the exact same AIR version and plugins has the same issues. Our new app signed by another dev (M1 Macs) results in everyone (including me) being unable to install the app (might be a license/cert issue on his side though).

I always have to go through distriqts resign script or it fails to install for me too (including with .444).

There also seems to be more issues with the newest AIR (which has signing updated?). We are unable to build apps on M1 Macs with .444. I opened another issue for that though #799

List of extensions used (all pretty recent, but I can go through all versions if really needed)

    <extensions>
        <extensionID>com.distriqt.InAppBilling</extensionID>
        <extensionID>com.distriqt.facebook.Core</extensionID>
                <extensionID>com.distriqt.facebook.Share</extensionID>
                <extensionID>com.distriqt.Bolts</extensionID>
        <extensionID>com.distriqt.Application</extensionID>
        <extensionID>com.distriqt.Volume</extensionID>
        <extensionID>com.distriqt.Core</extensionID>
        <extensionID>com.distriqt.playservices.GCM</extensionID>
        <extensionID>com.distriqt.playservices.Base</extensionID>
        <extensionID>com.distriqt.playservices.Ads</extensionID>
        <extensionID>com.distriqt.CustomResources</extensionID>
        <extensionID>com.distriqt.MediaPlayer</extensionID>
        <extensionID>com.distriqt.ZipUtils</extensionID>
        <extensionID>com.distriqt.NetworkInfo</extensionID>
        <extensionID>com.distriqt.Firebase</extensionID>
        <extensionID>com.distriqt.firebase.Performance</extensionID>
        <extensionID>com.distriqt.firebase.RemoteConfig</extensionID>
        <extensionID>com.google.firebase.core</extensionID>
        <extensionID>com.google.protobuflite</extensionID>
        <extensionID>androidx.core</extensionID>
        <extensionID>androidx.appcompat</extensionID>
        <extensionID>androidx.browser</extensionID>
        <extensionID>androidx.cardview</extensionID>
        <extensionID>androidx.recyclerview</extensionID>
        <extensionID>androidx.multidex</extensionID>
        <extensionID>com.google.android.material</extensionID>
        <extensionID>com.google.android.datatransport</extensionID>
        <extensionID>com.google.dagger</extensionID>
        <extensionID>com.distriqt.Notifications</extensionID>
        <extensionID>com.distriqt.PushNotifications</extensionID>
        <extensionID>com.distriqt.playservices.AdsIdentifier</extensionID>
        <extensionID>com.distriqt.IDFA</extensionID>
        <extensionID>androidx.constraintlayout</extensionID>
        <extensionID>androidx.vectordrawable</extensionID>
        <extensionID>com.google.code.gson</extensionID>
        <extensionID>com.distriqt.Adverts</extensionID>
        <extensionID>com.distriqt.admob.FacebookAudience</extensionID>
        <extensionID>com.distriqt.admob.IronSource</extensionID>
        <extensionID>com.distriqt.admob.AppLovin</extensionID>
        <extensionID>com.distriqt.admob.UnityAds</extensionID>
                <extensionID>com.android.installreferrer</extensionID>
    </extensions>

Your file upload request is expired. I'll happily upload it tomorrow.

ajwfrost commented 3 years ago

Hi @ventr1x - thanks for that, if you're able to upload the non-working and working IPA files using the below link we'll take a look. If you're able to use .444 (and even better, using it on something that's not Big Sur) it would be helpful. https://transfer.harman.com/requests/xQdEoOLj3gP6Ge35KkntBj

thanks

ventr1x commented 3 years ago

I packed one of our apps with .444 (I did not replace any files, not needed anymore?), with and without resign and let some people test installing it (we are using installrapp.com most of the time).

I'm using Catalina 10.15.7 on a MacBook Pro 2018 with newest intellij Ultimate.

java version "1.8.0_181" Java(TM) SE Runtime Environment (build 1.8.0_181-b13) Java HotSpot(TM) 64-Bit Server VM (build 25.181-b13, mixed mode)

XCode 12.4 (12D4e)

I'm not pointing to a local iOS SDK.

We have interesting results (not working = no message at all on older iOS or "integrity could not be verified" on newer ones):

Without resigning .ipa:

iPhone XS Max; iOS 14.4.2 => NOT working iPhone 7; iOS 12.1.2 => NOT working iPhone 5S; iOS 12.5.2 => working iPhone 7 Plus; iOS 14.4.2 => working iPhone 6 Plus; iOS 14.4.2 => working iPhone 11; iOS 14.4.2 => working

Result with resigned .ipa

iPhone XS Max; iOS 14.4.2 => working iPhone 7; iOS 12.1.2 => NOT working iPhone 5S; iOS 12.5.2 => working iPhone 7 Plus; iOS 14.4.2 => working iPhone 6 Plus; iOS 14.4.2 => working iPhone 11; iOS 14.4.2 => working

I had another iPhone 11 Pro; iOS 14.4.2 not working on any try, but I can't reach him right now.

First three devices are mine (the others I can't access, home offices far away), I don't mind resetting/updating the iPhone 7 for testing.

I've uploaded two .ipa files.

ajwfrost commented 3 years ago

Thanks @ventr1x - so from a first look, there are a couple of differences here: 1) our signed binaries contain both sha-1 based and sha-256 based signatures, although the resigned one only contains sha-256.. (probably the sha-1 compatibility was omitted by codesign because it could see the minimum OS requirement..) 2) there's the 'archived-expanded-entitlements.xcent' file included in the re-signed one (put there by the re-signing script I believe?) - this is meant to be optional afaik 3) but the entitlements are also different between the two binaries which is odd.. we have the "beta-reports-active" value but it's set to false; this is omitted in the re-signed version. And the keychain-access-groups has lost the wildcard in the re-signed one, to include the full app ID there.

The MinimumOSVersion is set to 12.2 which presumably is why it doesn't work on the iPhone 7 at 12.1.2.

But quite why this isn't working on the XS I'm not sure! We'll have to try a few things out and see if we can reproduce this here..

thanks

ventr1x commented 3 years ago

The issue is really killing any productivity for us. We are also unable to debug anything on device because of it. We are always getting hit with code signing errors:

WARNING: could not locate iTunesMetadata.plist in archive!
WARNING: could not locate Payload/x.app/SC_Info/x.sinf in archive!
ERROR: Install failed. Got error "ApplicationVerificationFailed" with code 0xe8008016: Failed to verify code signature of /var/installd/Library/Caches/com.apple.mobile.installd.staging/temp.W9fWxM/extracted/Payload/x.app : 0xe8008016 (The executable was signed with invalid entitlements.)

We are not adding anything custom to entitlements; Uploading it to app store as production does work without issues.

Don't forget that we also have a iPhone 11 Pro which cannot install ANY AIR apps from us right now (only development). Which - of course - is a client, which now is unable to test his own app for months.

ajwfrost commented 3 years ago

@ventr1x I'm not 100% sure whether these will help since the failure there is about the entitlements; I'm assuming you're not trying to install locally on a build that was created for the app store though?

In case it helps, we have an ADT version that will use codesign directly on a macOS machine: https://transfer.harman.com/message/xVZYjaeTvNAuS6XzVahuYy

There may be some open issues with this, so if you are still having problems, we can perhaps take a look at the IPA files. All the provisioning profile, entitlements and certificate device list still need to be set up of course..

One last thought, just to double-check: if you are having problems on macOS, then you should always be able to run the re-signing script that Distriqt provided, or follow the process mentioned under #511..? If that's not working, then there may be more fundamental issues that need to be addressed.. our goal of course would be to make sure you don't have to do any post-processing like this!

thanks

ventr1x commented 3 years ago

I'm only using development builds (not adhoc) and never touched any entitlements (they're set to development and the resign script is changing them to production when needed).

We are using resign for about 2 years now because of such issues (mostly because of Fb SDK). Just "recently" we get the issue of some devices not being able to install at all, no matter if resigned or not. I did not change or update my Mac, besides needed dev tools. I created all certificates on the same device I'm working on.

The resign script is not really helpful when trying to run on device for debugging. I haven't tried adding it as a pre-launch script in intellij though.

ajwfrost commented 3 years ago

@ventr1x with the 33.1.1.533 build, this should now fix things also on Windows, so I believe that this issue is now fixed.

I'm closing it for now but if you're able to check and see whether you still have a problem with signing or installing the signed IPA, please re-open and we can examine the IPA to see what may have gone wrong...

Thanks

idanasher commented 2 years ago

I am also having signing issues with air 33.1.1686 for ios.

I taught that with rather new version I won't have to resign and other signing issues, but when I try to upload via Transporter I get:

ERROR ITMS-90034: "Missing or invalid signature. The bundle 'com.facebook.sdk.FBSDKCoreKit' at bundle path 'Payload/Rummy World.app/Frameworks/FBSDKCoreKit.framework' is not signed using an Apple submission certificate."

I have then tried to manually resign using district script, but I got this in the resign console: ..../Frameworks/FBSDKCoreKit.framework: invalid or unsuppoted format for signature

and of course that the upload resolves in another signing issue. what am I doing wrong here ?

I'm focusing on 64-bit device only now I have upgrade my mac ios to: macOS Monterey 12.0.1 I have upgrade my xcode to : 13.1

marchbold commented 2 years ago

@idanasher we have responded in the other issue.