DexFileFactory.writeDexFile produces invalid files

[weary-adventurer]

Compile and run sample program that uses DexRewriter without any passes:

import com.android.tools.smali.dexlib2.DexFileFactory;
import com.android.tools.smali.dexlib2.dexbacked.DexBackedDexFile;
import com.android.tools.smali.dexlib2.iface.DexFile;
import com.android.tools.smali.dexlib2.rewriter.DexRewriter;
import com.android.tools.smali.dexlib2.rewriter.RewriterModule;

public class Main {
    public static void main(String[] args) {
        try {
            DexBackedDexFile dexFile = DexFileFactory.loadDexFile("classes_original.dex", null);
            DexRewriter rewriter = new DexRewriter(new RewriterModule());
            DexFile newDexFile = rewriter.getDexFileRewriter().rewrite(dexFile);
            DexFileFactory.writeDexFile("classes_rewritten.dex", newDexFile);
        }
        catch (Exception e) {
            e.printStackTrace();
        }
    }
}

Using smali library 3.0.7:

$ ls lib
smali-3.0.7.jar  smali-baksmali-3.0.7.jar  smali-dexlib2-3.0.7.jar  smali-util-3.0.7.jar guava-33.2.1-jre.jar  jcommander-1.64.jar 

Try to disassemble classes_original.dex (success):

$ java -cp "lib\*" com.android.tools.smali.baksmali.Main dis -o classes_original_out classes_original.dex

Try to disassemble classes_rewritten.dex (failure):

$ java -cp "lib\*" com.android.tools.smali.baksmali.Main dis -o classes_rewritten_out classes_rewritten.dex

Error occurred while disassembling class LFC0; - skipping class
com.android.tools.smali.util.ExceptionWithContext: Invalid uleb128 integer encountered at offset 0x110642
        at com.android.tools.smali.dexlib2.dexbacked.DexReader.readUleb128(DexReader.java:147)
        at com.android.tools.smali.dexlib2.dexbacked.DexReader.readSmallUleb128(DexReader.java:118)
        ...
Error occurred while disassembling class LVL1; - skipping class
com.android.tools.smali.util.ExceptionWithContext: Invalid string index 1237382, not in [0, 19974)
        at com.android.tools.smali.util.ExceptionWithContext.withContext(ExceptionWithContext.java:54)
        at com.android.tools.smali.baksmali.Adaptors.MethodDefinition.<init>(MethodDefinition.java:168)
        ...
Error occurred while disassembling class Landroidx.appcompat.widget.AppCompatTextView; - skipping class
com.android.tools.smali.util.ExceptionWithContext: Invalid string index 157401990, not in [0, 19974)
        at com.android.tools.smali.util.ExceptionWithContext.withContext(ExceptionWithContext.java:54)
        at com.android.tools.smali.baksmali.Adaptors.MethodDefinition.<init>(MethodDefinition.java:168)
        ...

Download: classes_original.dex

[melcz]

Hi, could you share the changes on rewriting that happen between the first and second disassembly attempts? Preferably in a minimal code snippet to try to reproduce the error on our end.

[weary-adventurer]

There are no changes between loading the file and rewriting it. Simply rewriting a file without changes using DexRewriter and then writing the rewritten file produces the broken dex. The code snippet that does it is listed as the first thing in the issue.

Here are all the files needed to reproduce it locally: smali_repro.zip.

Steps:

1) Download and extract smali_repro.zip

Libraries used

- https://maven.google.com/com/android/tools/smali/smali/3.0.7/smali-3.0.7.jar - https://maven.google.com/com/android/tools/smali/smali-baksmali/3.0.7/smali-baksmali-3.0.7.jar - https://maven.google.com/com/android/tools/smali/smali-dexlib2/3.0.7/smali-dexlib2-3.0.7.jar - https://maven.google.com/com/android/tools/smali/smali-util/3.0.7/smali-util-3.0.7.jar - https://repo1.maven.org/maven2/com/google/guava/guava/33.2.1-jre/guava-33.2.1-jre.jar - https://repo1.maven.org/maven2/com/beust/jcommander/1.64/jcommander-1.64.jar

2) Compile repro program: javac -cp ".;lib\*" Main.java (Windows) javac -cp ".:lib/*" Main.java (Linux)

3) Run repro program (it reads classes_original.dex and writes broken classes_rewritten.dex) java -cp ".;lib\*" Main (Windows) java -cp ".:lib/*" Main (Linux)

4) Disassemble original file classes_original.dex (no error): java -cp ".;lib\*" com.android.tools.smali.baksmali.Main dis -o out_original classes_original.dex (Windows) java -cp ".:lib/*" com.android.tools.smali.baksmali.Main dis -o out_original classes_original.dex (Linux)

5) Disassemble broken file classes_rewritten.dex (error as the file is broken): java -cp ".;lib\*" com.android.tools.smali.baksmali.Main dis -o out_rewritten classes_rewritten.dex (Windows) java -cp ".:lib/*" com.android.tools.smali.baksmali.Main dis -o out_rewritten classes_rewritten.dex (Linux)

The classes_rewritten.dex file is generated by using DexRewriter and then writing the file with DexFileFactory.writeDexFile.

[weary-adventurer]

Actually it looks like DexFileFactory.writeDexFile is broken.

Repro:

import com.android.tools.smali.dexlib2.DexFileFactory;
import com.android.tools.smali.dexlib2.dexbacked.DexBackedDexFile;

public class Main {
    public static void main(String[] args) throws Exception {
        DexBackedDexFile dexFile = DexFileFactory.loadDexFile("classes_original.dex", null);
        DexFileFactory.writeDexFile("classes_writedexfile.dex", dexFile);
    }
}

Compile and run:

$ javac -cp ".;lib\*" Main.java

$ java -cp ".;lib\*" Main

$ md5sum classes_original.dex
73d6b886ad21105a077ba4ff8dc70a83 *classes_original.dex

$ md5sum classes_writedexfile.dex
9b638d2cf2e768c87b44a4f7dd3f3e50 *classes_writedexfile.dex

$ stat -c "%s %n" -- *.dex
2938412 classes_original.dex
3172524 classes_writedexfile.dex

$ java -cp ".;lib\*" com.android.tools.smali.baksmali.Main dis -o out classes_writedexfile.dex
Error occurred while disassembling class LFC0; - skipping class
com.android.tools.smali.util.ExceptionWithContext: Invalid uleb128 integer encountered at offset 0x110642

[melcz]

Could you test with the changes in the rewriter branch? https://github.com/google/smali/tree/rewriter

[weary-adventurer]

It does not compile from command line. I've used both gradlew assemble and gradlew smali:fatJar and both fail. If you can build the jars like the ones I can download from maven then I'll be able to test it.

> Task :smali:generateGrammarSource FAILED
error(10):  internal error:  : java.lang.Error: Error parsing smaliTreeWalker.g: 'third_party' not expected 'grammar'
org.antlr.tool.GrammarSpelunker.match(GrammarSpelunker.java:70)
org.antlr.tool.GrammarSpelunker.grammarHeader(GrammarSpelunker.java:108)
org.antlr.tool.GrammarSpelunker.parse(GrammarSpelunker.java:80)
org.antlr.Tool.sortGrammarFiles(Tool.java:585)
org.antlr.Tool.process(Tool.java:439)
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
...
> Task :smali:generateGrammarSource FAILED

> Task :dexlib2:compileJava
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':smali:generateGrammarSource'.
> There were 19 errors during grammar generation

[melcz]

The changes have been merged, you can try with Smali 3.0.8 from the Maven repository.

[weary-adventurer]

It looks like there are no errors but the written file does not correspond to the original. It's much larger and it's missing static initializers for false and null. Is this intended?

Repro:

import com.android.tools.smali.dexlib2.DexFileFactory;
import com.android.tools.smali.dexlib2.dexbacked.DexBackedDexFile;

public class Main {
    public static void main(String[] args) throws Exception {
        DexBackedDexFile dexFile = DexFileFactory.loadDexFile("classes_original.dex", null);
        DexFileFactory.writeDexFile("classes_writedexfile.dex", dexFile);
    }
}

Use classes_original.dex from the first repro as input Produces classes_writedexfile.dex as output

Hashes:

$ md5sum *.dex
73d6b886ad21105a077ba4ff8dc70a83 *classes_original.dex
38a21bcc83ce194985a6234d75fcee46 *classes_writedexfile.dex

Compile:

$ javac -cp ".;lib\*" Main.java
$ java -cp ".;lib\*" Main

Using libraries:

$ md5sum lib/*
872309e5982530bdc7e68096c0d53cd2 *lib/guava-33.2.1-jre.jar
f45d6fa1223a2a2fb022277fed7b545a *lib/jcommander-1.64.jar
ebe46aaa23784c41752aca942222772d *lib/smali-3.0.8.jar
710f1dd2921360244ddc1a156665ff9b *lib/smali-baksmali-3.0.8.jar
6776c6a7fd5ccecee677dc327634f39a *lib/smali-dexlib2-3.0.8.jar
6bdfc62df7cf9bea9402b0e51c82c8e9 *lib/smali-util-3.0.8.jar

Disassemble both:

java -cp ".;lib\*" com.android.tools.smali.baksmali.Main dis -o out_classes_original.dex classes_original.dex
java -cp ".;lib\*" com.android.tools.smali.baksmali.Main dis -o out_classes_writedexfile.dex classes_writedexfile.dex

Compare disassembly:

$ diff -arq out_classes_original.dex out_classes_writedexfile.dex
Files out_classes_original.dex/ck1.1.smali and out_classes_writedexfile.dex/ck1.1.smali differ
Files out_classes_original.dex/MO4.smali and out_classes_writedexfile.dex/MO4.smali differ
Files out_classes_original.dex/TA0.smali and out_classes_writedexfile.dex/TA0.smali differ
Files out_classes_original.dex/Xj3.1.smali and out_classes_writedexfile.dex/Xj3.1.smali differ

Differences for ck1:

--- out_classes_original.dex/ck1.1.smali
+++ out_classes_writedexfile.dex/ck1.1.smali
@@ -4,9 +4,9 @@

 # static fields
-.field public static a:Z = false
+.field public static a:Z

-.field public static b:Z = false
+.field public static b:Z

 .field public static final c:Ljava/util/concurrent/atomic/AtomicBoolean;

Static initializers for original ck1:

No static initializers for rewritten ck1:

[UltraSina]

hi i have been noticed this too , do you finded any solation?

[UltraSina]

The changes have been merged, you can try with Smali 3.0.8 from the Maven repository.

its fixed in 3.0.8? mr @melcz