[QUESTION] Merge (conflicting) dex files

[auermich93]

Hi @Lanchon,

assume I need to add a JAR as a dependency to an existing APK. The first thing is to convert it to a .dex file using dx. That's fine, but I cannot simply place the .dex file alongside the original classes.dex file(s) and re-build the APK. I suppose the main dex file encodes some information to support multiple dex files. The current workaround is to convert both dex files to their smali representation and merge those folders manually and re-build the APK afterwards. That only works as long as there is enough space in the main .dex file. I thought about using multidexlib2, which can inherently handle multi-dex, but what happens when a potential conflict arises between those .dex files? It could happen that certain classes appear multiple times, because they are real duplicates or are different versions of the same class. Is multidexlib2 able to handle such conflicts on a best effort basis?

Thanks in advance!

[Lanchon]

I need to add a JAR as a dependency to an existing APK.

why? it would not be used. what else are you doing? what are you trying to accomplish? always ask for help to your root problem, not to problems found in solutions you think are best for the root problem.

I suppose the main dex file encodes some information to support multiple dex files.

it doesn't. (on ART and APKs for minSDK set to android 5 (first with ART).)

I cannot simply place the .dex file alongside the original classes.dex file(s) and re-build the APK.

yes you can. but obviously the dex versions have to match. but since you are using dx/d8 yourself, you choose the dex version of the converted jar.

convert both dex files to their smali representation and merge those folders manually

you can, but you gain nothing.

That only works as long as there is enough space in the main .dex file.

smali is a tool to work on dex files. if used in the context of multidex, you have to run smali separately for each dex.

I thought about using multidexlib2

again, simply adding the file does the trick

otherwise you can combine the dex/mdex easily by using "dexpatcher -o outdex dex1 dex2 dex3 ..."

however....

a multidex cannot have multiple definitions of a type (class, etc). (IMO) the ART verifier should reject such an mdex (haven't tested). most alternatives would depend on dex file processing order which is horrible. multidexlib2 correctly rejects an mdex with dup types. and i hope dexlib2 rejects dex files with dup types. and dexpatcher will also refuse to produce an invalid dex/mdex with dup types.

It could happen that certain classes appear multiple times, because they are real duplicates or are different versions of the same class. Is multidexlib2 able to handle such conflicts on a best effort basis?

there is no best effort possible. dup types means malformed dex: reject.

if you tell me what you really want to achieve and why you want to add a jar to an apk that it wont use anyway, i will probably be able to help you in a more meaningful way.

[auermich93]

I need to add a JAR as a dependency to an existing APK.

why? it would not be used. what else are you doing? what are you trying to accomplish? always ask for help to your root problem, not to problems found in solutions you think are best for the root problem.

OK, let me briefly summarize what I intend to do. I want to check whether certain runtime permissions are set, because it seems like sometimes they get dropped unexpectedly. Thus, I followed the recommendations and inserted more or less:

ContextCompat.checkSelfPermission(context, permission) == PERMISSION_GRANTED;

This requires the androidX library as a dependency. Since certain apps don't declare this dependency, I do need to insert it on my own. I originally thought this is a 'soft' dependency and a stub implementation is sufficient, but this is not the case.

I suppose the main dex file encodes some information to support multiple dex files.

it doesn't. (on ART and APKs for minSDK set to android 5 (first with ART).)

OK, that should actually simplify things a bit.

I cannot simply place the .dex file alongside the original classes.dex file(s) and re-build the APK.

yes you can. but obviously the dex versions have to match. but since you are using dx/d8 yourself, you choose the dex version of the converted jar.

OK, then the error message was misleading (couldn't resolve class ...) and probably the problem you described here appeared. That's why I thought that information about multi-dex is somehow encoded in the main dex file.

I thought about using multidexlib2

again, simply adding the file does the trick

otherwise you can combine the dex/mdex easily by using "dexpatcher -o outdex dex1 dex2 dex3 ..."

OK, I guess under the hood dexpatcher is using something like:

java -cp dx.jar com.android.dx.merge.DexMerger output.dex input1.dex input2.dex

a multidex cannot have multiple definitions of a type (class, etc). (IMO) the ART verifier should reject such an mdex (haven't tested). most alternatives would depend on dex file processing order which is horrible. multidexlib2 correctly rejects an mdex with dup types. and i hope dexlib2 rejects dex files with dup types. and dexpatcher will also refuse to produce an invalid dex/mdex with dup types.

It could happen that certain classes appear multiple times, because they are real duplicates or are different versions of the same class. Is multidexlib2 able to handle such conflicts on a best effort basis?

there is no best effort possible. dup types means malformed dex: reject.

OK, sounds legit.

[Lanchon]

I followed the recommendations and inserted more or less

i have to guess stuff because you are not explicit. i suppose you are roundtripping to smali and editing? (it would probably be easier to do that in dexpatcher but there is a steeper learning curve, so i dont recommend it for such a simple task.)

Thus, I followed the recommendations and inserted more or less: ContextCompat.checkSelfPermission(context, permission) == PERMISSION_GRANTED;

it seems you don't know why the compat libs are there in android. just use: context.checkSelfPermission(permission) == PERMISSION_GRANTED; increase the minSDK to 23 (android 6), and be done with it. you don't need to add any libraries.

Since certain apps don't declare this dependency, I do need to insert it on my own.

probably every app you'll encounter includes a version of the compat libs. problem is, the lib is probably obfuscated and so the names of the classes and methods have been changed. or at least the APK was trimmed down and unused methods were removed. note that google's lib might not work if imported multiple times in a single APK. also, some lib names were probably not obfuscated due to keep rules, and those will conflict with a second import of the code. no AI systems exist to my knowledge to solve these issues automatically.

I guess under the hood dexpatcher is using something like...

absolutely not

[auermich93]

I followed the recommendations and inserted more or less

i have to guess stuff because you are not explicit. i suppose you are roundtripping to smali and editing? (it would probably be easier to do that in dexpatcher but there is a steeper learning curve, so i dont recommend it for such a simple task.)

I am sorry, typically people blame you when you start telling stories. I do instrument branches in the bytecode, which in turn (when being reached) call a custom supplied Tracer class, which in turn writes to the external storage. Logs indicated that writing failed in some cases due to a permission issue. Thus, my idea was to insert the permission check in order to debug it. In the meantime, I could resolve the issue (those runtime permissions were dropped due to some adb command in the testing framework). So you are right, dexpatcher is no option here, since it is just for debugging.

Thus, I followed the recommendations and inserted more or less: ContextCompat.checkSelfPermission(context, permission) == PERMISSION_GRANTED;

it seems you don't know why the compat libs are there in android. just use: context.checkSelfPermission(permission) == PERMISSION_GRANTED; increase the minSDK to 23 (android 6), and be done with it. you don't need to add any libraries.

To be honest, I have no idea why people require this compat libs, I just followed the workflow suggested by the developer guide: https://developer.android.com/training/permissions/requesting

Since certain apps don't declare this dependency, I do need to insert it on my own.

probably every app you'll encounter includes a version of the compat libs. problem is, the lib is probably obfuscated and so the names of the classes and methods have been changed. or at least the APK was trimmed down and unused methods were removed. note that google's lib might not work if imported multiple times in a single APK. also, some lib names were probably not obfuscated due to keep rules, and those will conflict with a second import of the code. no AI systems exist to my knowledge to solve these issues automatically.

Yeah, you are right. The app I used for testing solely contained the minimum required part of the compat library, lacking the ContextCompat class. And yeah, one should keep in mind those obfuscated methods, they can screw up everything.

Just out of curiousity, how is your dexpatcher actually merging two methods? Is it creating a new method or trying to merge the patch in place? In the latter case, how do you handle those register restrictions? Are there any stats how successful dexpatcher is? Is the merged code guaranteed to be working, i.e. not causing any verification errors?

[auermich93]

Gonna check it myself.

[Lanchon]

sorry i never saw this