search

anestisb / android-prepare-vendor

Set of scripts to automate AOSP compatible vendor blobs generation from factory images
347 stars 155 forks source link

Angler (Nexus 6p) API-26 support #84

Closed anestisb closed 6 years ago

anestisb commented 7 years ago
poshul commented 6 years ago

I've submitted a first pass at "Create full configuration" in #94

poshul commented 6 years ago

I can confirm that the vendor files generated compile with the latest copperhead. However CNEService crashes constantly: AndroidRuntime: java.lang.VerifyError: Verifier rejected class com.quicinc.cne.CNE$CNEReceiver: void com.quicinc.cne.CNE$CNEReceiver.run() failed to verify: void com.quicinc.cne.CNE$CNEReceiver.run(): [0xFA] Cannot infer method from invoke-virtual-quick (declaration of 'com.quicinc.cne.CNE$CNEReceiver' appears in /system/priv-app/CNEService/CNEService.apk)

thestinger commented 6 years ago

Pixels don't strip out classes.dex so these issues haven't been a problem there. Nexus 5X/6P are messier.

anestisb commented 6 years ago

Xmm that's not good because the ART runtime seems to complain about an optimised method. Since N6p is stripping bytecode it probably means that something is not working as expected when repairing the bytecode.

Are you using the default oatdump repair method or the optional baksmali (--smali flag)?

poshul commented 6 years ago

I'm using the default, I can try with --smali and let you know how that works.

poshul commented 6 years ago

SMALI crashes outright.

anestisb commented 6 years ago

Seems that latest baksmali is not happy with parsed odex files.

$ java -jar /home/anestisb/Developer/android/android-prepare-vendor/hostTools/Java/baksmali.jar x -o /tmp/android_img_repair.3Zl8nb/atfwd/deopt -d /tmp/android_img_repair.3Zl8nb/arm64 /home/anestisb/Developer/android/android-prepare-vendor/bullhead/opr6.170623.013/factory_imgs_data/system/priv-app/atfwd/oat/arm64/atfwd.odex

Error occurred while loading class path files. Aborting.
java.lang.ArrayIndexOutOfBoundsException: 795111833
        at org.jf.dexlib2.dexbacked.BaseDexBuffer.readSmallUint(BaseDexBuffer.java:53)
        at org.jf.dexlib2.dexbacked.DexBackedClassDef.<init>(DexBackedClassDef.java:78)
        at org.jf.dexlib2.dexbacked.DexBackedDexFile$1.readItem(DexBackedDexFile.java:137)
        at org.jf.dexlib2.dexbacked.DexBackedDexFile$1.readItem(DexBackedDexFile.java:133)
        at org.jf.dexlib2.dexbacked.util.FixedSizeSet$1.next(FixedSizeSet.java:56)
        at org.jf.dexlib2.analysis.DexClassProvider.<init>(DexClassProvider.java:48)
        at org.jf.dexlib2.analysis.ClassPathResolver.loadEntry(ClassPathResolver.java:241)
        at org.jf.dexlib2.analysis.ClassPathResolver.loadLocalOrDeviceBootClassPathEntry(ClassPathResolver.java:210)
        at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:120)
        at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:105)
        at org.jf.baksmali.AnalysisArguments.loadClassPathForDexFile(AnalysisArguments.java:129)
        at org.jf.baksmali.AnalysisArguments.loadClassPathForDexFile(AnalysisArguments.java:86)
        at org.jf.baksmali.DisassembleCommand.getOptions(DisassembleCommand.java:207)
        at org.jf.baksmali.DeodexCommand.getOptions(DeodexCommand.java:71)
        at org.jf.baksmali.DisassembleCommand.run(DisassembleCommand.java:181)
        at org.jf.baksmali.Main.main(Main.java:102)

Ok I'll look into it starting with the oatdump bytecode repair method.

poshul commented 6 years ago

There's a discussion of this bug here: https://github.com/JesusFreke/smali/issues/498

anestisb commented 6 years ago

The included jars match tag 2.2.1 so the mentioned fix should be part the used version. So probably something is missing or changed when Oreo production libart was released.

poshul commented 6 years ago

Looking further into the changes in O, it appears that there is a new vdex file which is generated during the dex2oat process. The vdex file has the uncompressed dex code in it (along with a header and footer). Would it be possible to have system-img-repair just use the vdex files where available instead of de-oating?

anestisb commented 6 years ago

Well assuming that vdex maintains the original dex (not sure what the "pre-validated" term means in the CL) we need a vdex parser. I don't see any standalone utils in AOSP so we probably have to write one from scratch.

I had a quick look in vdex format and it appears to be a relatively simple format. I'm guessing the only tricky part would to handle multi-dex files.

Will start writing a parser PoC if my oatdump analysis doesn't lead anywhere. I prefer to fix an existing method rather than creating a new one. I've already done this exercise 3-times (smaliex, smali & oatdump) and I'm not a big fun of another one that needs extensive testing across all supported devices (except Pixel which don't strip bytecode).

poshul commented 6 years ago

For what its worth, I hand stripped the CNEService vdex down to the contained dex, updated the checksum, and manually made an unoptimized apk. The OS still throws the same java.lang.VerifyError.

Hopefully that saves you from having to write yet another parser.

anestisb commented 6 years ago

I'm not surprised since the upstream AOSP oatdump utility has been updated to support vdex format. As such when we use the '--export-to' to dump the DEX from OAT behind the scenes it dumps from the VDEX since the bytecode is no longer present in the OAT ELF file. At least this is how I read the code from AOSP.

Since the VDEX appears to implement DEX-to-DEX (quickened) transformations (thus why you had to repair checksum) we're still in the same problematic situation of relying on other tools to resolve vtable and repair all quickened opcodes. With API-25 we're lucky since the ART was supporting already optimized methods, but it seems that this is not the case anymore with API-26.

If baksmali doesn't resolve the problem not sure what we can do to fix this. Since ART supports runtime optimising due to JIT hybrids there might be some hidden logic we can use. However, I doubt that any of them will also revert DEX-to-DEX transformations.

poshul commented 6 years ago

Looking at vdex.h https://android.googlesource.com/platform/art/+/master/runtime/vdex_file.h There is a method to fully unquicken the dex based on metadata stored in the vdex file. I'm going to mess around with writing a tool that can at least unquicken and extract single dex vdex files. I'll let you know if it goes anywhere.

anestisb commented 6 years ago

It would be awesome if we can get an offline unquicken utility. However, seeing the method dependencies to DEX decompiler seems like a lot of work.

Since oatdump links to libart it might be easier to just invoke unquickening from inside oatdump code. Ideally we can upstream that feature too since it would come handy to many people.

poshul commented 6 years ago

I spent some time working on adding a switch to oatdump to pull the dex files from the vdex and unquickening them. There is method included in vdexfile called unquicken. So far the modified program keeps segfaulting. I will keep working on it

anestisb commented 6 years ago

For the record the same issue affects bullhead Oreo support too (https://github.com/anestisb/android-prepare-vendor/issues/85).

thestinger commented 6 years ago

I'll just do a release without all of this stuff which I had to do last year. None is particularly critical: It'll break RCS, CNE (fancy Qualcomm transparent LTE <-> WiFi switches that are smarter than AOSP alone, which doesn't exist on non-Qualcomm devices anyway), restoring time on boot (which has been broken for various periods of time in the past anyway).

thestinger commented 6 years ago

Ah actually TimeService is in vendor for the Nexus 6P, it'll only break time restoring for the Nexus 5X.

poshul commented 6 years ago

Sounds good. I've been running with everything else working for several days and I don't see degraded performance (of the phone at least, my performance has been degraded trying to find why VdexFile::Unquicken segfaults constantly.)

On Fri, Sep 15, 2017 at 1:37 PM, Daniel Micay notifications@github.com wrote:

Ah actually TimeService is in vendor for the Nexus 6P, it'll only break time restoring for the Nexus 5X.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/anestisb/android-prepare-vendor/issues/84#issuecomment-329849015, or mute the thread https://github.com/notifications/unsubscribe-auth/AAI9i_p0AC076k4SFv-snefWgQi3TJ5Dks5sirXLgaJpZM4PQ51Q .

poshul commented 6 years ago

So I think the issue with unquicken is related to a problem mentioned in https://android.googlesource.com/platform/art/+/de4b08ff24c330d5b36b5c4dc8664ed4848eeca6%5E%21/#F8

Namely that LEB encoding causes problems for subsequent 16bit indices. Which would explain why the crash was happening for me in the instruction after unpacking a 128 LE value.

anestisb commented 6 years ago

I've modified the oatdump sources to include a dex unquickening logic prior to exporting the DEX files from the VDEX that matches the optimised OAT. Evidence so far indicate that it resolves the problem via the art host binaries, although I want to do some more testing.

The plan is to update the oatdump binary deps that are used by the project to include the unquickening patch. I'll also clean-up the code and submit to upstream AOSP so we can have the code merged with next major Android release. I'm attaching the patch file in case you want to do some additional testing from your side.

unquicken.patch.txt

anestisb commented 6 years ago

Did some more extensive testing with 6p using the naked config and the bytecode repair issues appear to be resolved. I've updated the oatdump linux dep links and added some logic to auto-pick the updates.

N5x testing is planned for the next days.

anestisb commented 6 years ago

Nexus 6p support appears to have a stable enough state to close the master issue. Specific issues can be reported as a separate bug.