search

JesusFreke / smali

smali/baksmali
6.31k stars 1.07k forks source link

ArrayIndexOutOfBoundsException in 8.1 SystemUIGoogle #597

Open MatiHalperin opened 6 years ago

MatiHalperin commented 6 years ago

I'm trying to decompile SystemUIGoogle.odex from walleye-opm1.171019.019-factory-b9946689.zip and I'm getting:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1889914952
    at org.jf.dexlib2.dexbacked.BaseDexBuffer.readSmallUint(BaseDexBuffer.java:53)
    at org.jf.dexlib2.dexbacked.OatFile$DexEntryIterator.next(OatFile.java:579)
    at org.jf.dexlib2.dexbacked.OatFile$DexEntryIterator.next(OatFile.java:570)
    at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
    at org.jf.util.AbstractForwardSequentialList$1.next(AbstractForwardSequentialList.java:91)
    at java.util.AbstractSequentialList.get(AbstractSequentialList.java:88)
    at org.jf.dexlib2.DexFileFactory.loadDexFile(DexFileFactory.java:129)
    at org.jf.baksmali.DexInputCommand.loadDexFile(DexInputCommand.java:144)
    at org.jf.baksmali.DisassembleCommand.run(DisassembleCommand.java:161)
    at org.jf.baksmali.Main.main(Main.java:102)

The commands I used were to move everything in /system/framework to /tmp/framework and:

java -jar baksmali-2.2.2.jar x -d /tmp/framework/arm64 SystemUIGoogle.odex

peteraldous commented 6 years ago

I'm having the same issue with Android 8.1 odex files. I've attached one that I've built from AOSP in case it's useful for debugging. If I find anything useful digging into this, I'll follow up.

Never mind. I can't upload arbitrary file types. If it would be useful to you, please let me know and I'll send it to you some other way.

peteraldous commented 6 years ago

I spent quite some time trying (without success) to get this program to build in an IDE. As we all know, Java debugging with jdb is reserved for the sixth circle of Dante's inferno. If anyone can comment with a way to get a reasonable debug system working with this program, I'll spend some more time digging into this error. For now, I'll have to move on to other things and hope this works itself out.

JesusFreke commented 6 years ago

I do all development on this in IDEA.

It's been a while since I set it up from scratch, but try ./gradlew build followed by ./gradlew idea and then open the generated project in idea

JesusFreke commented 6 years ago

(This was obviously all done before IDEA had such nice built-in gradle support.. :) )

peteraldous commented 6 years ago

In case this is useful to anyone, building in the Windows Subsystem for Linux and then opening in IDEA does not work. However, building in Windows with gradlew.bat (with JDK 8; JDK 10 didn't work) as @JesusFreke suggested does seem to work.

I'm out of time for messing with this at the moment but I'll try to dig back in.

peteraldous commented 6 years ago

After a little digging around, it seems to actually be reading a crazy-high number from the ByteStream buffer using readSmallUint(). The offset it reads indicates where next to read - but the number read is orders of magnitude higher than the size of the buffer. It may be that the odex file format has changed.

Since I couldn't upload the odex file that was giving me trouble, I forked the repository and added the file to a new directory called debug. You can find the forked repo here: https://github.com/peteraldous/smali

peteraldous commented 6 years ago

Here's another breadcrumb: in the odex file I added to the forked repository in the last comment, the number being read is the size of the key/value store. However, the offset is calculated incorrectly. In this line from vim's hex representation of the file, we see the relevant data:

00001040: ca65 8e53 00b0 a570 ee08 0000 636c 6173 .e.S...p....clas

The integer read as the size of the k/v store is 0x70a50b00 (located at 0x00001044; int values are little-endian). The actual size of the store is 0x000008ee, which is the next word in the file (located at 0x00001048). I have not found any relevant bits of information in the documentation that indicate why the disassembler would be off by one word. This may be because dex is documented but odex isn't.

JesusFreke commented 6 years ago

Yes, the oat format changes fairly regularly. e.g. see oat_file.h (iirc) in AOSP.

I haven't had a chance to see what changes, if any are present for 8.1.

peteraldous commented 6 years ago

I believe the relevant difference is these lines from oat_file.h:

// Pointer to the beginning of the ArtMethods in .bss section, if present, otherwise null. uint8_t bssmethods;

An additional pointer in the OatFile class would explain the additional word. I need to work on other things but I have some more time I'll try to dig into the smali code and try to figure out how to fix it in a stable and robust way.

JesusFreke commented 6 years ago

Yeah, you'll need to find what oat version that change was introduced in. The CL that introduced that change should have incremented the oat version in oat.h (see: kOatVersion).

And then, in OatFile.java, you can use getOatVersion() to change the parsing logic depending on the oat version of the file being read. See, e.g. DexEntryIterator.next()

peteraldous commented 6 years ago

I don't suppose anyone has an Android 8.0 odex file lying around? I'm new to reverse engineering file formats and a point of reference would be useful. It'll take a while to build my own, as the only way I've found to do it is to rebuild all of Android...

The most useful point of reference would be one of the odex files generated in out/target/product/system/framework. I've been using am.odex.

JesusFreke commented 6 years ago

The easiest way is to grab one of the firmwares from https://developers.google.com/android/images and extract the system image. You can use a utility called "simg2img" to convert the system image file to a mountable image, and then loop mount it on linux.

peteraldous commented 6 years ago

Hrm the current version of baksmali chokes on the odex file I pulled out of that image, but at a different point. I'll see if I can fix this specific issue but it looks like there may be other things to figure out.

peteraldous commented 6 years ago

In case it's useful:

Error occurred while loading class path files. Aborting. org.jf.dexlib2.analysis.ClassPathResolver$ResolveException: org.jf.dexlib2.analysis.ClassPathResolver$NotFoundException: Could not find classpath entry boot.oat at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:145) at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:105) at org.jf.baksmali.AnalysisArguments.loadClassPathForDexFile(AnalysisArguments.java:129) at org.jf.baksmali.AnalysisArguments.loadClassPathForDexFile(AnalysisArguments.java:86) at org.jf.baksmali.DisassembleCommand.getOptions(DisassembleCommand.java:207) at org.jf.baksmali.DeodexCommand.getOptions(DeodexCommand.java:71) at org.jf.baksmali.DisassembleCommand.run(DisassembleCommand.java:181) at org.jf.baksmali.Main.main(Main.java:102) Caused by: org.jf.dexlib2.analysis.ClassPathResolver$NotFoundException: Could not find classpath entry boot.oat at org.jf.dexlib2.analysis.ClassPathResolver.loadLocalOrDeviceBootClassPathEntry(ClassPathResolver.java:216) at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:120) ... 7 more

Fatmajk commented 6 years ago

I have the same issue. I am not able to decode any .odex files. I launch this command:

sudo java -jar baksmali.jar x -c framework/arm/boot.oat -d framework/ Messages.odex -o out/ or sudo java -jar baksmali.jar x Messages.odex

It gives me this error:

Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 1890058312
    at org.jf.dexlib2.dexbacked.BaseDexBuffer.readSmallUint(BaseDexBuffer.java:53)
    at org.jf.dexlib2.dexbacked.OatFile$DexEntryIterator.next(OatFile.java:579)
    at org.jf.dexlib2.dexbacked.OatFile$DexEntryIterator.next(OatFile.java:570)
    at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
    at org.jf.util.AbstractForwardSequentialList$1.next(AbstractForwardSequentialList.java:91)
    at java.util.AbstractSequentialList.get(AbstractSequentialList.java:88)
    at org.jf.dexlib2.DexFileFactory.loadDexFile(DexFileFactory.java:129)
    at org.jf.baksmali.DexInputCommand.loadDexFile(DexInputCommand.java:144)
    at org.jf.baksmali.DisassembleCommand.run(DisassembleCommand.java:161)
    at org.jf.baksmali.Main.main(Main.java:102)

I'm using baksmali (v.2.2.2) and Messages.odex from a Huawei P20 ROM.

Solution: I've found a solution to my problem. Decode the .vdex with VdexExtractor (found here: https://github.com/anestisb/vdexExtractor) and then use the latest dex2jar (works with Android 8+ since they have a new .dex format) (found here: https://github.com/DexPatcher/dex2jar/releases)

agorski3 commented 6 years ago

A pull request I made should fix issues with the Android 8.1 OAT format at least up to OAT (131) which is what AOSP uses.

It can be found here.

One other thing to note, they changed the kMultiDexSeparatorString to "!" instead of ":". This does not appear to affect the smali code but will affect anyone who attempts to parse the dex location string in a oat file containing multiple dex entries. Take boot-framework.oat for example. It always contains multiple dex. In Android 7.1, the entry for the location of the second dex file is "/system/framework/framework.jar:classes2.dex" where as 8.1 it is "/system/framework/framework.jar!classes2.dex". The change was made in OAT 130 and threw another wrench into some of my code so I figured I would mention it here in case it helps anyone else out.

peteraldous commented 6 years ago

Update:

Thanks to @agorski3 for contributing!

The new jar (2.2.4) behaves the same as the old one.

I pulled and built it myself and now it gives a different error (I use an alias for baksmali):

$ baksmali de am.odex -o am

Error occurred while loading class path files. Aborting.
org.jf.dexlib2.analysis.ClassPathResolver$ResolveException: org.jf.dexlib2.analysis.ClassPathResolver$NotFoundException: Could not find classpath entry boot.oat
        at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:145)
        at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:105)
        at org.jf.baksmali.AnalysisArguments.loadClassPathForDexFile(AnalysisArguments.java:129)
        at org.jf.baksmali.AnalysisArguments.loadClassPathForDexFile(AnalysisArguments.java:86)
        at org.jf.baksmali.DisassembleCommand.getOptions(DisassembleCommand.java:207)
        at org.jf.baksmali.DeodexCommand.getOptions(DeodexCommand.java:71)
        at org.jf.baksmali.DisassembleCommand.run(DisassembleCommand.java:181)
        at org.jf.baksmali.Main.main(Main.java:102)
Caused by: org.jf.dexlib2.analysis.ClassPathResolver$NotFoundException: Could not find classpath entry boot.oat
        at org.jf.dexlib2.analysis.ClassPathResolver.loadLocalOrDeviceBootClassPathEntry(ClassPathResolver.java:216)
        at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:120)
        ... 7 more

As before, I'm attempting to disassemble odex files I built from AOSP and then reassemble them to dex files.

agorski3 commented 6 years ago

Notice that 2.2.4 was built using code up to and including commit ec31f76. The commit with my changes in it is f10643f which is right after that one. So it is not included into 2.2.4 for some reason. Not sure why @JesusFreke decided not to include it in 2.2.4 but that was his decision. This is why it behaved like before for 2.2.4.

As for the new error, the error says it all "Could not find classpath entry boot.oat". So you are not providing the boot.oat file to baksmali when disassembling the odex file. In other words, the problem is how you are calling baksmali. You need to use the "-b boot.oat -d path/to/dir/containing_boot_oat" options.

JesusFreke commented 6 years ago

2.2.4 was released before your pull request was ready :)

peteraldous commented 6 years ago

Thanks to both of you. I'll dig around in my built AOSP and see if I can find a boot.oat file somewhere.

JesusFreke commented 6 years ago

But I'm planning on doing another 2.2.5 release soon :)

agorski3 commented 6 years ago

Ah that makes sense. No problem I just found it strange since both changes happened around the same time.

peteraldous commented 6 years ago

I found boot.oat in the output of the Android build I did to generate my odex files. It now gives a different (higher) number for the index that's out of bounds:

$ baksmali de am.odex -o am -b ~/bogus/boot.oat

Error occurred while loading class path files. Aborting.
java.lang.ArrayIndexOutOfBoundsException: 1919254169
        at org.jf.dexlib2.dexbacked.BaseDexBuffer.readSmallUint(BaseDexBuffer.java:53)
        at org.jf.dexlib2.dexbacked.DexBackedClassDef.<init>(DexBackedClassDef.java:77)
        at org.jf.dexlib2.dexbacked.DexBackedDexFile$1.readItem(DexBackedDexFile.java:147)
        at org.jf.dexlib2.dexbacked.DexBackedDexFile$1.readItem(DexBackedDexFile.java:143)
        at org.jf.dexlib2.dexbacked.util.FixedSizeSet$1.next(FixedSizeSet.java:56)
        at org.jf.dexlib2.analysis.DexClassProvider.<init>(DexClassProvider.java:48)
        at org.jf.dexlib2.analysis.ClassPathResolver.loadEntry(ClassPathResolver.java:241)
        at org.jf.dexlib2.analysis.ClassPathResolver.loadLocalClassPathEntry(ClassPathResolver.java:179)
        at org.jf.dexlib2.analysis.ClassPathResolver.loadLocalOrDeviceBootClassPathEntry(ClassPathResolver.java:191)
        at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:120)
        at org.jf.dexlib2.analysis.ClassPathResolver.<init>(ClassPathResolver.java:85)
        at org.jf.baksmali.AnalysisArguments.loadClassPathForDexFile(AnalysisArguments.java:135)
        at org.jf.baksmali.AnalysisArguments.loadClassPathForDexFile(AnalysisArguments.java:86)
        at org.jf.baksmali.DisassembleCommand.getOptions(DisassembleCommand.java:207)
        at org.jf.baksmali.DeodexCommand.getOptions(DeodexCommand.java:71)
        at org.jf.baksmali.DisassembleCommand.run(DisassembleCommand.java:181)
        at org.jf.baksmali.Main.main(Main.java:102)