Storyyeller / Krakatau

Java decompiler, assembler, and disassembler
GNU General Public License v3.0
1.95k stars 219 forks source link

bytecode peephole optimization non-looping-loops #186

Open Janmm14 opened 2 years ago

Janmm14 commented 2 years ago

In the original obfuscation the long j is not set to 0, but I can easily edit the bytecode so that its a 0.

long j = 0L;
long j0 = 3154942220328664823L;
while((j ^ j0) != -4097245258586862040L) {
    {
        j = -8987102950668834246L;
        j0 = 4928288597787943954L;
    }
}

I think it should be possible to optimize these never-looping-loops relatively easy, but I'm struggling with such generic bytecode analysis. bytecode of sample above:

L10:    lconst_0 
L11:    ldc2_w 3154942220328664823L 
L14:    goto L23 

        .stack full 
            [redacted]
        .end stack 
L17:    ldc2_w -8987102950668834246L 
L20:    ldc2_w 4928288597787943954L 

        .stack full 
            [redacted]
        .end stack 
L23:    lxor 
L24:    ldc2_w -4097245258586862040L 
L27:    lcmp 
L28:    ifne L17 

other sample of similar never looping thing:

long j = 0L;
long j0 = 6913978580415874324L;
while(true) {
    int i = (j < j0) ? -1 : (j == j0) ? 0 : 1;
    if (i == 0) {
        break;
    }
    if (i != -1) {
        j = 1371682301368612379L;
        j0 = 98363365008051593L;
    } else {
        // real code
        break;
    }
}

185

Edit: maybe that whole thing is also in another non-looping-loop:

label0: while(true) {
    long j1 = 0L;
    long j2 = 5954778881033184416L;
    while(true) {
        int i0 = (j1 < j2) ? -1 : (j1 == j2) ? 0 : 1;
        if (i0 == 0) {
            break;
        }
        if (i0 == -1) {
            break label0;
       }
        j1 = 234223452657458715L;
        j2 = 6736292361240488430L;
    }
}
KOLANICH commented 2 years ago

I think deobfuscation is out of scope of Krakatau, its killer feature is robustness and fairness, so it can decompile even non-java langs compiled into JVM bytecode.

I remember there used to be plenty of Java deobfuscators.

Storyyeller commented 2 years ago

Loop unrolling would be straightforward to do in principle - the problem is that most of the time it would make things worse. You basically need a way to "guess" when to do it.

Janmm14 commented 2 years ago

Those deobfuscators, one of which I'm also maintaining a little bit, usually don't have such advanced peephole optimizations and often rely on bytecode patterns. I thought that maybe it'd be possible for Krakatau to handle this part because it does other optimizations already. I guess loop unrolling would need to be tried in some branch and then some detection if it removes the loop or so in the end.

But I can accept if this is deemed out-of-scope. In that case I'll need to find some motivation to improve my understanding of ow2's asm analyzer at some point. I mean the code between or inside these loops is readable, so its just an annoyance.

Storyyeller commented 2 years ago

By the way, do you know if there are any examples of this obfuscation in a freely available app? Having a sample to test with would be really helpful.

Janmm14 commented 2 years ago

https://github.com/java-deobfuscator/deobfuscator/issues/748 At least I couldn't find it being sold or offered for download somewhere when I googled so I gave that the benefit of the doubt that he followed the rule set on the issues there which is to not infringe copyright.

Edit: This is the jar I took above bytecode from: deob03.zip

Col-E commented 2 years ago

By the way, do you know if there are any examples of this obfuscation in a freely available app?

At least I couldn't find it

Bozar is open source. You can find some samples of it along with others on SkidSuite's obfuscation demo samples.