android N preview npd56n (Vendor generation failed)

[lightydo]

It doesn't work at all if oat2dex.jar is not updated to the latest version (compiled directly from git tree)

sudo ./execute-all.sh -d bullhead -b npd56n -o tmp -i ~/Downloads/bullhead-npd56n-factory-996cac57.tgz [sudo] password for danield: [] Setting output base to 'tmp/bullhead/npd56n' [] Extracting 'bullhead-npd56n-factory-996cac57.tgz' [] Unzipping 'image-bullhead-npd56n.zip' [] Copying files from system partition ... [] Copying files from vendor partition ... [] Preparing environment for 'arm' ABI [] Preparing environment for 'arm64' ABI [] Start extracting system partition & de-optimize pre-compiled bytecode ... [-] '/priv-app/GoogleDialer/GoogleDialer.apk' de-optimization failed consider manual inspection - skipping archive [_] '/priv-app/PrebuiltGmsCore/PrebuiltGmsCore.apk' not pre-optimized with sanity checks passed - copying without changes [-] '/priv-app/CNEService/CNEService.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/ExternalStorageProvider/ExternalStorageProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/CalendarProvider/CalendarProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/Shell/Shell.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/GooglePartnerSetup/GooglePartnerSetup.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/GCS/GCS.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/ConnMO/ConnMO.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/Phonesky/Phonesky.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/DMService/DMService.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/TeleService/TeleService.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/DCMO/DCMO.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/DiagMon/DiagMon.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/CellBroadcastReceiver/CellBroadcastReceiver.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/TelephonyProvider/TelephonyProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/BlockedNumberProvider/BlockedNumberProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/SprintDM/SprintDM.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/SetupWizard/SetupWizard.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/GooglePackageInstaller/GooglePackageInstaller.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/EmergencyInfo/EmergencyInfo.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/ConnMetrics/ConnMetrics.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/MtpDocumentsProvider/MtpDocumentsProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/Settings/Settings.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/SettingsProvider/SettingsProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/ConfigUpdater/ConfigUpdater.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/MediaProvider/MediaProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/WallpaperCropper/WallpaperCropper.apk' de-optimization failed consider manual inspection - skipping archive [-] 'tmp/bullhead/npd56n/factory_imgsdata/system/priv-app/CtsShimPrivPrebuilt/CtsShimPrivPrebuilt.apk' not pre-optimized & without 'classes.dex' - skipping [-] '/priv-app/CarrierEntitlement/CarrierEntitlement.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/Velvet/Velvet.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/GoogleLoginService/GoogleLoginService.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/ContactsProvider/ContactsProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/HotwordEnrollment/HotwordEnrollment.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/GoogleContacts/GoogleContacts.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/DownloadProvider/DownloadProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/GoogleOneTimeInitializer/GoogleOneTimeInitializer.apk' de-optimization failed consider manual inspection - skipping archive [] '/priv-app/GoogleServicesFramework/GoogleServicesFramework.apk' not pre-optimized with sanity checks passed - copying without changes [-] '/priv-app/StatementService/StatementService.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/ManagedProvisioning/ManagedProvisioning.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/SystemUI/SystemUI.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/TagGoogle/TagGoogle.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/LifeTimerService/LifeTimerService.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/Telecom/Telecom.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/MmsService/MmsService.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/Entitlement/Entitlement.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/DocumentsUI/DocumentsUI.apk' de-optimization failed consider manual inspection - skipping archive [-] '/priv-app/GoogleFeedback/GoogleFeedback.apk' de-optimization failed consider manual inspection - skipping archive [-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/ext.jar' not pre-optimized & without 'classes.dex' - skipping [-] '/framework/com.google.android.media.effects.jar' de-optimization failed consider manual inspection - skipping archive [-] '/framework/services.jar' de-optimization failed consider manual inspection - skipping archive [-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/okhttp.jar' not pre-optimized & without 'classes.dex' - skipping [-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/org.apache.http.legacy.boot.jar' not pre-optimized & without 'classes.dex' - skipping [-] 'tmp/bullhead/npd56n/factory_imgsdata/system/framework/telephony-common.jar' not pre-optimized & without 'classes.dex' - skipping [] Skipping framework-res since it doesn't pair with bytecode [-] '/framework/android.test.runner.jar' de-optimization failed consider manual inspection - skipping archive [-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/ims-common.jar' not pre-optimized & without 'classes.dex' - skipping [-] '/framework/com.google.android.maps.jar' de-optimization failed consider manual inspection - skipping archive [-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/voip-common.jar' not pre-optimized & without 'classes.dex' - skipping [-] 'tmp/bullhead/npd56n/factory_imgsdata/system/framework/core-libart.jar' not pre-optimized & without 'classes.dex' - skipping [-] '/framework/pm.jar' de-optimization failed consider manual inspection - skipping archive [] '/framework/rcsservice.jar' not pre-optimized with sanity checks passed - copying without changes [-] '/framework/javax.obex.jar' de-optimization failed consider manual inspection - skipping archive [-] 'tmp/bullhead/npd56n/factory_imgsdata/system/framework/bouncycastle.jar' not pre-optimized & without 'classes.dex' - skipping [] '/framework/rcsimssettings.jar' not pre-optimized with sanity checks passed - copying without changes [-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/core-junit.jar' not pre-optimized & without 'classes.dex' - skipping [-] '/framework/uiautomator.jar' de-optimization failed consider manual inspection - skipping archive [-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/framework.jar' not pre-optimized & without 'classes.dex' - skipping [-] 'tmp/bullhead/npd56n/factory_imgsdata/system/framework/apache-xml.jar' not pre-optimized & without 'classes.dex' - skipping [] '/framework/cneapiclient.jar' not pre-optimized with sanity checks passed - copying without changes [-] '/framework/monkey.jar' de-optimization failed consider manual inspection - skipping archive [-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/core-oj.jar' not pre-optimized & without 'classes.dex' - skipping [-] 'tmp/bullhead/npd56n/factory_imgs_data/system/framework/conscrypt.jar' not pre-optimized & without 'classes.dex' - skipping [-] '/framework/wifi-service.jar' de-optimization failed consider manual inspection - skipping archive [-] '/app/BasicDreams/BasicDreams.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/GoogleHome/GoogleHome.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/GoogleContactsSyncAdapter/GoogleContactsSyncAdapter.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/UserDictionaryProvider/UserDictionaryProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Maps/Maps.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/RCSBootstraputil/RCSBootstraputil.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Photos/Photos.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/PrebuiltExchange3Google/PrebuiltExchange3Google.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/GoogleCamera/GoogleCamera.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Hangouts/Hangouts.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/CalculatorGooglePrebuilt/CalculatorGooglePrebuilt.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/YadaYada/YadaYada.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/YouTube/YouTube.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/talkback/talkback.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Tycho/Tycho.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/PartnerBookmarksProvider/PartnerBookmarksProvider.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/PlayGames/PlayGames.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/FaceLock/FaceLock.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/GoogleJapaneseInput/GoogleJapaneseInput.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/GoogleVrCore/GoogleVrCore.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/GoogleHindiIME/GoogleHindiIME.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/PrebuiltDeskClockGoogle/PrebuiltDeskClockGoogle.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/KoreanIME/KoreanIME.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/HiddenMenu/HiddenMenu.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/HTMLViewer/HTMLViewer.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/CalendarGooglePrebuilt/CalendarGooglePrebuilt.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Chrome/Chrome.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Wallet/Wallet.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/NfcNci/NfcNci.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/GooglePinyinIME/GooglePinyinIME.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/PrebuiltGmail/PrebuiltGmail.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/GoogleTTS/GoogleTTS.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/WebViewGoogle/WebViewGoogle.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/CloudPrint2/CloudPrint2.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/RcsImsBootstraputil/RcsImsBootstraputil.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Bluetooth/Bluetooth.apk' de-optimization failed consider manual inspection - skipping archive [-] 'tmp/bullhead/npd56n/factory_imgsdata/system/app/CtsShimPrebuilt/CtsShimPrebuilt.apk' not pre-optimized & without 'classes.dex' - skipping [-] '/app/PrebuiltBugle/PrebuiltBugle.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Videos/Videos.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Drive/Drive.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/DMAgent/DMAgent.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Music2/Music2.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/LatinIMEGooglePrebuilt/LatinIMEGooglePrebuilt.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/PrintSpooler/PrintSpooler.apk' de-optimization failed consider manual inspection - skipping archive [-] '/app/Stk/Stk.apk' de-optimization failed consider manual inspection - skipping archive [] System partition successfully extracted & de-optimized at 'tmp/bullhead/npd56n/factory_imgs_repaireddata' [!] Target device expects to have following img versions when using output system img [] Booatloder:BHZ11d [] Baseband:M8994F-2.6.33.2.08 [] Generating blobs for vendor/lge/bullhead [*] Copying files to 'tmp/bullhead/npd56n/vendor/lge/bullhead' ... cp: cannot stat ‘tmp/bullhead/npd56n/factory_imgs_repaired_data/system/app/atfwd/atfwd.apk’: No such file or directory [-] Vendor generation failed

[anestisb]

Thanks for reporting.

Usually I don't spent time with preview images since they're not supposed be fully functional and thus pose hidden challenges that result into more (usually unnecessary) work. Will have a look when possible to see if something has changed in the way bytecode is pre-optimized or the blobs list is different for Android-N.

Have you tried to flash the vanilla image in a Nexus 5x phone to verify all hardware stacks (GSM, WiFi, NFC, Bluetooth, etc.) are working?

[lightydo]

When you say vanilla you mean AOSP? So far I have been able to build the Android N preview 3 for Nexus 5x but it won't boot (boot loop...related to keystore?). The preview 3 stock image works fine. Since there are no instructions with regards to building and flashing Android N AOSP on Nexus 5x I can;t figure out what to do...

[thestinger]

The preview sources aren't fully open. The tags don't correspond to the code shipped by Google, at least for many of the repositories.

[anestisb]

When I mentioned vanilla I was referring to AOSP using the blobs from the preview channel (https://developers.google.com/android/nexus/blobs-preview). From the perspective of the vendor blobs extraction script we're interested at first point to identify possible additions/deletions that will affect the extraction list from the /system partitions (priv-apps, libs, etc.).

But it seems that we don't have corresponding tags yet so building is not a straight forward options yet, leaving us only with the factory preview image to work with.

Digging into the factory preview image it seems that the blobs at which your run failed are still there. So probably something went wrong in the extraction process. I'll start digging the OAT version that Android N is using at the latest image and ensure that de-optimization tools are compatible with the corresponding libart version.

Will keep you posted here.

[anestisb]

SmaliEx used to de-optimize the OAT files doesn't seem to support (yet) the libart version used in Android N preview images. If you comment the &>/dev/null parts in system-img-repair.sh script you will see the various errors when de-optimizing the OAT files.

Hopefully it will get fixed by the time Android N reaches production, otherwise we'll have to migrate to a different tool (not sure if a decent alternative exists) for the de-optimization of OAT files, or deal with them in a different way at the build process (e.g. patchoat them to align with AOSP build boot.oat and include them as pre-optimized bytecode).

I'll leave the bug open for now until we have any final updates / decisions.

[thestinger]

It appears that SmaliEx is able to cope with it now: https://github.com/testwhat/SmaliEx/commit/227f79ca11a29979f2cb34d69aa0be0ed17e857f.

[anestisb]

@thestinger thanks for the update. Had an early look at the Nexus 9 (WiFi) Android N factory image and seems that Google is no longer pushing apps pre-optimized. The only pre-compiled oat files present are the framework jars. We might actually get lucky here and avoid the de-optimization headache for the Android N releases.

Will wait for the Nexus 5x/6p factory images to appear online to verify this new policy before start coding around the Android N support. Fingers crossed we might actually have less work to be handled here. I just hope that the proprietary blobs list is not changed for the same devices. Have some concerts about the bluetooth modules since I've noticed a recent big refactor at the AOSP tree.

[thestinger]

There's definitely some new stuff in /system. They seem to have replaced with pittpatt blobs in /vendor with different blobs there. I'm not sure which stuff can be considered to be part of Google Play though.

[thestinger]

I'm having a lot of trouble simply getting a working build for the Nexus 9 and it doesn't really seem to be related to blobs... it simply doesn't mount /vendor for some reason. Mounting it manually on a userdebug build does work. Bonus: upgrades from 6.0.1 -> 7.0 are broken due to the new encryption stuff.

[anestisb]

Then probably something changed at the build chain and the script generated makefiles are missing pieces (or broken). Unfortunately we'll have to do lots of digging/tweaking work since I just noticed the Nexus 5x NRD90M image has different policy from N9 and the system apps are pre-optimized.

I'll work with it during the weekend and keep you posted with available options (probably at the CopperheadOS IRC channel). Will start with N9 since deps are significantly smaller compared to N5x/N6p.

[thestinger]

It turns out the issue is that it can't find the verity metadata for /vendor. It might have actually been a problem before but verity is actually enforcing now. I'm not sure.

[thestinger]

They apparently only optimize frameworks on the Nexus 9 due to extremely bad I/O performance, where the optimized code takes longer to load. Nexus 6 might be similar.

[anestisb]

Have started working compatibility with Android N. First success device was Nexus 9 WiFi (tested with a user-debug build and so far no issue). Have created separate issues per device to track down progress. Closing this one since preview images are no longer a thing.