DexPatcher / dexpatcher-tool

Android Dalvik bytecode patcher.
https://dexpatcher.github.io/
GNU General Public License v3.0
428 stars 79 forks source link

Add options for "targetPosition" #13

Closed gurt-il closed 6 years ago

gurt-il commented 7 years ago

Hi

I'm using your tools a lot for many applications. Recently, I've tried to use the tool for an app that has been compiled with proguard. As part of the "proguard" process, the files names are changing to a, aa ant etc. The packages names are also changes to a, aa and etc.

I'm trying to patch a file that is under the package "com.app.c". But, The app also contains (after the "proguard" process) class named "com.app.c" and I'm getting the error "Error:(1, 1) error: package com.app.c clashes with class of same name".

After many many tries, I've figured that the only way to solve this problem is if you will add a "targetPosition" options to the tag. This will allow me to patch the code in different place (like "com.app.c_") and then to tell the dexpatcher to put this code on "com.app.c".

Many Many thanks for all the great work (and for adding the life-saving "multi-dex" option) Ysrael

Lanchon commented 7 years ago

hi!

proguard's a bitch! :)

i'm afraid i don't fully understand the problem, you'll have to explain it better, and preferably with code. you'll need to use precise words please.

I'm trying to patch a file that is under the package "com.app.c".

file? you mean a class? what is the full name of the class? something like com.app.c.theClassName?

But, The app also contains (after the "proguard" process) class named "com.app.c"

you mean class c in package com.app? and do you care about this class too? do you want to patch it? do you want to access it from your patch code?

and I'm getting the error "Error:(1, 1) error: package com.app.c clashes with class of same name".

where do you get this error? i assume it comes from javac? i assume you are using the gradle plugin in android studio? then the plugin is feeding javac with a jar containing the symbols from the app, so you can reference them without declaring them.

i didn't know javac couldn't shadow a class in a library with a new package. this is very strange, because it can certainly shadow a class in a library with new class. this might be a bug in javac.

After many many tries, I've figured that the only way to solve this problem is if you will add a "targetPosition" options to the tag. This will allow me to patch the code in different place (like "com.app.c_") and then to tell the dexpatcher to put this code on "com.app.c".

there are many, many improvements i would like to implement in dexpatcher. unfortunately i dont have the time, no sponsors, and there are very few users of dexpatcher. at first i thought a dexpatcher would catch a lot of users, just like xposed did, but noone seems to like it too much... :(

anyway, dexpatcher already has the target= argument in DexEdit that can be used for this. unfortunately renaming a class is more complex than simply changing its name: instructions in dex assembly might reference fields, methods, etc from the class, and your code compiled by javac must be rewritten to work within the renamed class. also, there is an interesting interaction with onlyEditMembers=true. all this was planned for a future release, but to tell you the truth i don't know if it will happen. i have way to many ideas for dexpatcher and very little time.

anyway. back to your problem.

the gradle plugin is not prepared to handle proguarded apks. ideally one would provide a map file to rename the interesting parts of the apk (only packages, classes and members that you care about). during build. the apk symbols would be transformed using this map before being fed to javac. so in your patch code you would only reference and edit stuff using "human" names. during build, after running javac on your patch, the map would be used to reverse-map your compiled patch to match the proguarded names of the apk, and only after that would dexpatcher be invoked.

this means that if you want to work on a new version the source apk (with completely different proguarded names), you wouldn't need to touch your code! you would only provide a new map file and rebuild. (and yes, tools could be developed to aid in the production of the new map based on the old one.)

but this is all if i had the time... i don't.

so what are your options now? you have several...

use target=

you would do something like this:

package hack.app;

@DexEdit(target = "com.app.c.theClassName", onlyEditMembers = true)
class theClassNameHook {
    @DexReplace
    int theMethod(int arg) {    // <-- PROBLEM HERE !!!
        // don't code here!!
        // this code would need to be rewritten by dexpatcher to be correct.
        // to be safe, just invoke a static method somewhere else:
        return theClassNameHandler.theMethod(arg);
    }
}

//@DexAdd
public class theClassNameHandler {
    public static int theMethod(int arg) {
        // your code here...
    }
}

seems nice but... see the "PROBLEM HERE !!!" line.

unfortunately theMethod is not static and takes a this pointer as its first argument. and the this pointer is of type hook.app.theClassNameHook. when the patch method is plugged into the apk by dexpatcher, the method it replaces would receive a this pointer of type com.app.c.theClassName, and this causes two problems. 1) a verification error would be raised on runtime because com.app.c.theClassName is not convertible to hook.app.theClassNameHook. but that won't happen, because 2) dexpatcher would notice the discrepancy at patch time and wouldn't let you patch a method with an incompatible signature (it would correctly complain that the target method was not found).

so... no go?

it depends...

package hack.app;

@DexEdit(target = "com.app.c.theClassName", onlyEditMembers = true)
class theClassNameHook {
    @DexReplace
    private static int thePrivateInstanceMethod(com.app.c.theClassName self, int arg) {
        // don't code here!!
        return theClassNameHandler.thePrivateInstanceMethod(self, arg);
    }
}

//@DexAdd
public class theClassNameHandler {
    public static int thePrivateInstanceMethod(com.app.c.theClassName self, int arg) {
        // your code here...
    }
}

manually edit the apk library

normally you setup an android studio project with two subprojects: one builds an apk library from an apk. the other produces the patched app based on the apk library and your code.

but there is no reason why these processes must be done "online" with each other. you can split the dependency: in the patched subproject gradle script file, remove the dependency from the source subproject, and instead manually add the 'TheApp.apk.aar' as you would add any '.aar' dependency in android studio.

once you have this working, edit the '.apk.aar' apk library file: inside it (it's a zip file) you will find a jar with the symbols from the apk. remove the offending "com/app/c.class" file from the jar, reassemble the '.apk.aar' file with the new jar, and you are good to go. of course, you wont be able to reference the problematic class 'c' in your patch at all.

use dex2jar toolset to rename the patch classes

you can create a "patch library" (see the appropriate dexpatcher sample project) placing your patch code in package 'com.app.c'. then manually process the code inside the resulting patch library with dex2jar. it has a tool to rename classes, use it to rename package 'com.app.c' to 'com.app.c'. then add the manually processed patch library to the patched subproject.

this sucks because of manual intervention on each build, but you could integrate the step into your gradle build. (but integration will take you much more time than you imagine, i promise! :) )


so all this happens because dexpatcher can't rewrite the method signatures and code of classes. yes, i have a solution for this thought out, but i'm just lazy about implementing it. if anything, all this shows that even simple stuff that dexpatcher handles is more complex below the surface than you'd suppose. this is a good thing, i want dexpatcher to hide most of the complex stuff.

so tell me how it goes!

Lanchon commented 7 years ago

btw, i want to add that handling cases like these is part of a bigger roadmap in dexpatcher that allows patching of anonymous inner classes, and that allows a template patch class to patch several real classes. the necessary pieces to solve all these puzzles are very similar, with only a query mini-language to find inner classes by -say- superclass type needed specifically to allow comfortable patching of anonymous inner classes.

Lanchon commented 7 years ago

any news?

gurt-il commented 7 years ago

Following you suggestions, I decided to create a dedicated tool that will be able to map the obfuscated code to more readable code. The tool is written in C#, and currently, he can rename packages, classes, and methods. You need a manual map once, and then the tool trying to migrate to newest versions automatically base on statistic data like who used the variable method, number of static methods on class and etc.

The result is surprisingly accurate, and now I can work with the obfuscated code easily almost like the regular code.

Currently, the code is not ready to release (too many manual fixes) but when it will be ready, I will release it to public.

And thanks again Lanchon, You are the BEST

Lanchon commented 7 years ago

wow! ive been meaning to write that for ages now :)

so i guess the target= trick wasn't enough for you, too bad. anyway, i was considering adding the code rewriter so that targeted class edits would just work, mostly because you needed it.

one caveat about renaming: much code uses reflection, and renaming kills reflection. that's why my idealized workflow would be:

source.dex --rename-> descrambled.dex --dex2jar-> symbols.jar
symbols.jar,patch.java --javac-> patch.classes --dx-> patch.dex --inverse-rename-> scrambled-patch.dex
source.dex,scrambled-patch.dex --dexpatcher-> patched.dex

this means that the rename is only used to create the symbols against which to compile the patch code, but not to produce the output. then the patch is inverse-renamed to obfuscate it matching the source dex, and finally dexpatcher is run against the obfuscated versions of both the source and patch, to produce and obfuscated patched dex.

note that the dex2jar toolset already can rename stuff using a map file. it works only on jars, but the above idealized workflow can be trivially altered to adapt it (swap rename and dx/dex2jar order).

however, it doesn't do any kind of rename-map porting to new versions of obfuscated bytecode.

Lanchon commented 6 years ago

hi,

just wanted to let you know that i published not one but two fixes for this issue this week:

thanks! :)

gurt-il commented 6 years ago

Thanks!