search

rovo89 / Xposed

The native part of the Xposed framework (mainly the modified app_process binary).
Other
7.38k stars 1.47k forks source link

Working on bringing ART support to Xposed... #18

Closed adramalech closed 9 years ago

adramalech commented 9 years ago

Hey,

I am a software engineer that is curious about contributing to this project. I was wanting to start contributing to creating a solution for ART support. I have pulled down Google/platform-art, Xposed, Xposed-Bridge, and Xposed-Installer. I started reading through the code for Xposed trying to understand . I was wondering where there is any documentation on setting up a debugging/testing environment so I could be able to step through the code and see how it interacts. If you have the time I would love maybe some further information so I can get a good starting point. Furthermore, I have read into ART but again feeling there isn't a lot of documentation for how ART works and interacts from a developers perspective.

Any tips, tricks, books, talks, documentation, etc. that would help would be greatly appreciated. Thanks!

rovo89 commented 9 years ago

Missing documentation is indeed a key problem with ART... The code has very few comments, and what's more important, there doesn't seem to be any architecture blueprint or something like that, which would describe on a higher level which parts ART consists of and how they work together.

Meanwhile, there seem to be a few articles that give a bit of an overview: https://www.youtube.com/watch?v=EBlTzQsUoOw http://www.anandtech.com/show/8231/a-closer-look-at-android-runtime-art-in-android-l https://plus.google.com/u/0/+MatthiasSchaff/posts/U1jhMAv3d7R https://plus.google.com/u/0/+MatthiasSchaff/posts/WGXSXv2Aukr

I didn't watch/read these myself as they were published long after I started researching myself. Unfortunately, none of them really goes into detail enough. The only chance you have is to read the source code. I don't think there's a way to step through it, you have to follow the code... You should probably checkout the complete AOSP source tree, then you can compile ART and add some debug messages.

Some hints about the basic flow:

Also interesting: Several Java classes have a "mirror" in native code, so actions on either side a closely linked.

I hope this helps to give you a start.

adramalech commented 9 years ago

Hey thank you soo much Rovo89! Will indeed help. I hope to get started asap... I did watch both the Google I/O 2014 event with the ART release talk. Then read a couple different g+ articles, and release articles from tech blogs on how ART works. So yeaa I pretty much already read all those links, but I thank you for the time showing me that. The most difficult part is the overhauling about the platform-art/compiler and platform-art/runtime/gc switching from JIT to AOT compilers and the GC has been updated mark and sweep from Dalvik.

Going to pull down AOSP source tree and just use good 'ol printf/cout... lol :-)

sunwayforever commented 9 years ago

I have written a demo that could support a minimal xposed plugin running on android ART. But it's only a demo, I have no time to dive into ART to solve a lot of problems, and the demon ONLY support hook compiled method of non preloaded classes. so, I must remove ActivityThread from preloaded-classes for this demo to work, also, I must make some workaround in stack.cc of ART to resolve native crash when dumping stack stace in java.

the demo is at https://github.com/sunwayforever/xposed-art

rovo89 commented 9 years ago

Nice. ;) Good work! And I see you're messing around with the stack and assembler, so you got a good impression of ART's complexity already.

I had tried a similar approach (among many others), but gave up on it for different reasons (e.g. different stack layouts for other platforms, very high complexity and risk for errors). I totally understand what you mean with "lot of problems" and the native crash, I went through this as well. It's kind of frustrating when you got a PoC working, but then come across a different case which totally invalidates your achievements...

I finally came to the conclusion that I have to replace libart.so and libart-compiler.so, which has several advantages:

I'm not ready to publish my source code yet. I will do so once I got so far that I can also publish the binaries for alpha testing. Then I will ping you, so you can check out my approach.

sunwayforever commented 9 years ago

Thanks for your information, Indeed I am feeling totally frustrated when dealing with the stack layout myself, but it's really a great experience for me to understand ART (or it's complexity...)... please let me know if any progress.

awakened1712 commented 9 years ago

@sunwayforever how did you "remove ActivityThread from preloaded-classes for this demo to work"? Is there a way to hook methods from preloaded classes?

awakened1712 commented 9 years ago

@rovo89 how to "disable some optimizations" in order to make it possible to hook all methods?

rovo89 commented 9 years ago

I have modified the ART source code and recompiled libart-compiler.so. The details are too complex to mention here. It's not possible to do this via configuration file or something.

samashok commented 9 years ago

Oic. The ART has source code for modifying I see. What about SIM, RIM and HARDWARE INTERRUPT? the instructions which do not have operand part & they operate on the contents of the accumulator only. LXI H, 2500H: Load HL Pair with 2500H COMPARE operation of these induction except carry, Auxiliary carry reset flag are updated. How to find Machine Cycles of an instruction? Caz in immediate addressing mode the operand is specified within the instruction itself. For this instruction the operand is in the memory. The address of the memory is not directly given in the instruction. The address of the memory resides in H-L pair & this has already been specified by an earlier instruction in the program, I.e. LXI H, 2500H. 3E, 05: The instruction in the code form.

IN O2: Read data from port C. Is this instruction O2 be the address of the port C? Of an I/O port from where the data is to be read.? Examples are one-byte instruction.

rovo89 commented 9 years ago

The ART has source code for modifying I see.

Modify what? Sorry, I didn't understand a word of what you wrote...

evilwombat commented 9 years ago

Here's a crazy thought. Rather than trying to patch the code at runtime, would it be easier to patch the contents of the APK being installed, at install time? That is, to scan the APK being installed for calls to methods we want to hook, and replace these with calls to stub methods that we control?

Sadly, my messing around with android has been quite limited, so please excuse me if the above does not make technical sense.

kmark commented 9 years ago

I'm not ready to publish my source code yet. I will do so once I got so far that I can also publish the binaries for alpha testing.

Have you considered pushing out your progress every so often? Might be faster/easier to collaborate on a few branches rather than everyone working on their own private implementation. I understand that from a support and code quality standpoint it's a pain in the ass but I think it'd be highly advantageous here.

mofirouz commented 9 years ago

From my very limited understanding:

@evilwombat what you say does make sense, but i doubt it will make much difference. You need to understand that in pre-5.0 Xposed doesn't actually make any changes to the bytecode of the application - it only creates maps of method call names to memory addresses of certain application method signatures so they could be called when the original methods are called (along with a few other neat tricks methinks).

What you are saying is almost as hard as actually porting Xposed to Art - because you need to generate code / bytecode through ASM, which both must be what the intended original java code supposed to be, and what it will compile to once it goes through ART (never mind binary resources / layouts etc and linking them).

Crazy idea: Go the pure linux way - and find a way to determine how memory is allocated to those binaries, and where methods address locations are stored on RAM?

evilwombat commented 9 years ago

@mofirouz - Fair enough. I was kind of hoping we could create Xposed-specific wrappers for all the methods we would want to hook, and somehow bring them into the application's environment (Android's equivalent of an import table? Crazy stuff with LD_PRELOAD, if each app is its own process?). These wrapper methods would have arguments/signatures that would be identical to the methods they were meant to wrap. I thought this would allow us to patch the APK/Java bytecode to replace calls to Android methods with calls to our wrappers, without having to do any kind of fixups of the nearby code, and without having any deviation from the "intent" of the original code at the call site. Then the wrapper method could either call through to the method it's wrapping, or call into Xposed to do something more interesting. But, my knowledge of this stuff at the application level is obviously quite limited, and I will wait for whatever solution the community comes up with.

manvir-singh commented 9 years ago

I'm sure there's a resion for why we haven't done this already but i cant seem to think of it, so why not modify "framework.jar"?

pylerSM commented 9 years ago

modifying framework.jar or APKs is not solution, we can go back to smali hacking than.

I also agree that @rovo89 should maybe push his WIP code so other people don't have to reinvent wheel. People would see what is done, what is not ready and could focus on TODO things.

manvir-singh commented 9 years ago

@pylerSM we have the source of framework.jar or is it device specific. @pylerSM agreed @rovo89 should really push his WIP code.

evilwombat commented 9 years ago

If he doesn't feel comfortable pushing WIP code, let him finish it first. Some people have a coding preference that doesn't lend itself to early release. Let the man do what he is comfortable with...

mofirouz commented 9 years ago

I completely agree. We are here to come up with ideas for the benefit of him and the community. However at the end of the day, it is his project and we should respect and be thankful to him :) On 14 Jan 2015 20:46, "evilwombat" notifications@github.com wrote:

If he doesn't feel comfortable pushing WIP code, let him finish it first. Some people have a coding preference that doesn't lend itself to early release. Let the man do what he is comfortable with...

— Reply to this email directly or view it on GitHub https://github.com/rovo89/Xposed/issues/18#issuecomment-69988342.

rovo89 commented 9 years ago

Thanks guys. Actually, there is no need to worry about alternative approaches. It's not easy to "hack" ART in a similar way as Dalvik, but it's generally working.

What most people forget:

a) ART isn't the only new thing in Lollipop. SELinux requires a lot of extra code and even more thoughts to work around the restrictions. And 64 bit breaks the assumption that everything starts in a single process, as there are two Zygote processes now. I think it's important to consider this right from the start.

b) The actual (net) time required to write code is far away from the time that passes. It's not like I work on Xposed every evening. There are weeks when I don't look at the code at all. Yes, I do have other things in my life, and some have a higher priority. So even if something would take just a few evenings to code, it might take me a couple of weeks to actually do it. And once I publish something, you probably wouldn't like it if I took off for a few weeks.

If it was just about coding, pushing the source could be helpful. But that works best if there's already a clear concept of what is needed and an idea how this can be split up into work packages. Often it takes hours to come up with a concept that can be coded in 20 lines. Remember how many people are using Xposed meanwhile. We don't want to have some quick code that works by accident, so it's vital to think everything through carefully, and that takes time.

So yeah. It will take more time, I still don't give any ETA, but I'm quite confident that we'll have something nice some day.

schoudhary2014 commented 9 years ago

@rovo89 Thank you very much for your work & opening it for community. I am eagerly waiting for ART support.

Moto-x commented 9 years ago

I hope that in the nearest time .

Tragen commented 9 years ago

I hope you can do it. I'm waiting with my update to Lollipop until xposed is ready for ART. Is there anything we can do to support the development so it goes faster?

Moto-x commented 9 years ago

I stopped using xposed a long time ago. Always felt like it is affecting battery life. I hope in the new update to solve this problem :\

Tragen commented 9 years ago

Perhaps it uses more battery, but I prefer to pay with battery life instead of my data. ;)

theknut commented 9 years ago

So have you verified this behavior with all modules disabled? I'm 99% certain one (or more) of your modules have caused this.

Remember, since rovo did an excellent job in providing an easy interface every noob can write modules. I have had a look at every module I installed and I ended up rewriting some of the module for myself. Am 02.02.2015 17:29 schrieb "Moto-x" notifications@github.com:

I stopped using xposed a long time ago. Always felt like it is affecting battery life. I hope in the new update to solve this problem :\

— Reply to this email directly or view it on GitHub https://github.com/rovo89/Xposed/issues/18#issuecomment-72487247.

pylerSM commented 9 years ago

Yes, I think some module causes that. Try disable them one by one to find bad module. theknut, which modules did you rewrite? where we can find them?

theknut commented 9 years ago

I have written a module which has all the small tweaks in it. Things like disabling the empty battery alerts, always expanded notifications, no screenshot delay and so on. I didn't want to install a module for each tiny tweak so I took the idea and implemented it in one module. It's not public, it's only for me. Am 02.02.2015 18:31 schrieb "pyler" notifications@github.com:

Yes, I think some module causes that. Try disable them one by one to find bad module. theknut, which modules did you rewrite? where we can find them?

— Reply to this email directly or view it on GitHub https://github.com/rovo89/Xposed/issues/18#issuecomment-72499253.

phillipberndt commented 9 years ago

@moto-x @tragen @pylersm @theknut .. Could you please keep all of that unrelated talk out of here?! Github sends an email per comment to all subscribers, and while that is interesting for technical insights and updates from rovo89, it's really quite annoying when its just your general chatter. Thanks.

ragnatelato commented 9 years ago

@rovo89 because as you can not work on an ongoing basis to Xposed not make open source? so we also work the other and we're not all waiting for lollipop without news.

evilwombat commented 9 years ago

News is that he is making progress. Let the man work - pressure doesn't help in these situations. This stuff is hard (and you get what you paid for :) ).

Faslane commented 9 years ago

Yes...PLEASE don't release it open source. People simy want early access to it by asking this and (I'm sure) you won't even consider this. Please Don't. Awesome job Rovo! Much appreciated (and donated of course) +1

kmark commented 9 years ago

@Faslane Compiling Xposed is sufficiently complicated (specific build process is largely undocumented) and time consuming enough that it will deter most from finding the source immediately useful for obtaining early-access. I would further assume that most of the people who could get it working from source would be at some position, at least technically, to contribute. Quite a few people have voiced interest in contributing at various stages of the project and it's clear that it is simply not a part of rovo's current workflow.

With that said there are a number of problems when releasing source for high-demand pre-release software like Xposed. Namely being that rovo could be flooded with requests for building instructions or Xposed doesn't compile on my computer issues. Many created by people who would need their hand held to complete the process regardless. From what I understand rovo is pressed for time as-is and cannot spare additional hours for what is essentially tedious project management. The more time he can spend writing code, testing, etc. means a faster release cycle for Xposed.

While I cannot speak for everyone participating on this issue, claiming people simply want early binary access is highly presumptuous.

Faslane commented 9 years ago

I don't understand your point. Its exactly what I'm saying already. I've seen people request it who would in no way help at all and honestly think it'd be for early access and nothing more.

Cheerz On Feb 16, 2015 6:43 PM, "Kevin Mark" notifications@github.com wrote:

@Faslane https://github.com/Faslane Compiling Xposed is sufficiently complicated (specific build process is largely undocumented) and time consuming enough that it will deter most from finding the source immediately useful for obtaining early-access. I would further assume that most of the people who could get it working from source would be at some position, at least technically, to contribute. Quite a few people have voiced interest in contributing at various stages of the project and it's clear that it is simply not a part of rovo's current workflow.

With that said there are a number of problems when releasing source for high-demand pre-release software like Xposed. Namely being that rovo could be flooded with requests for building instructions or Xposed doesn't compile on my computer issues. Many created by people who would need their hand held to complete the process regardless. From what I understand rovo is pressed for time as-is and cannot spare additional hours for what is essentially tedious project management. The more time he can spend writing code, testing, etc. means a faster release cycle for Xposed.

While I cannot speak for everyone participating on this issue, claiming people simply want early binary access is highly presumptuous.

— Reply to this email directly or view it on GitHub https://github.com/rovo89/Xposed/issues/18#issuecomment-74609067.

Faslane commented 9 years ago

I see people ask for betas and early access daily and have to presume most won't participate in bettering it. Just my opinion and what I've seen. Carry on devs! :-)

kmark commented 9 years ago

@Faslane I was referencing only this specific developer-oriented GitHub thread and I assumed you were as well. I'm certain there's a good bit of begging going on over on XDA, reddit, and elsewhere. My apologies for the communication failure.

Faslane commented 9 years ago

Oh, got it. No and I may have posted incorrectly in here as well. Do over! Haha. +1 On Feb 16, 2015 7:00 PM, "Kevin Mark" notifications@github.com wrote:

@Faslane https://github.com/Faslane I was referencing only this specific developer-oriented GitHub thread and I assumed you were as well. I'm certain there's a good bit of begging going on over on XDA, reddit, and elsewhere. My apologies for the communication failure.

— Reply to this email directly or view it on GitHub https://github.com/rovo89/Xposed/issues/18#issuecomment-74610260.

DavisNT commented 9 years ago

ART inlines "easy"/short methods into the caller

@rovo89 Looks like very simple methods (e.g. return false;) still get inlined with Xposed for Lollipop alpha (xposed-arm-20150213b.zip).

rovo89 commented 9 years ago

@rovo89 Looks like very simple methods (e.g. return false;) still get inlined with Xposed for Lollipop alpha (xposed-arm-20150213b.zip).

Could you please show me an example? Ideally, it should be a small app. Check the .oat file in the Dalvik cache with "oatdump" to find out which native code the function (and the call to it) compiles to.

vladpolkovnik commented 9 years ago

@rovo89 Just in case you don't check the xposed.info website inbox; sent you some code that I have been working on. Let me know if I can help somehow.

DavisNT commented 9 years ago

@rovo89 Could you please show me an example? Ideally, it should be a small app.

I have posted an example Android app: https://github.com/DavisNT/XposedLollipopTests/ method2() gets inlined here. There is also another bug - sometimes when updating method2() and relaunching the app (without reboot) it crashes.

kmark commented 9 years ago

May be worth creating separate issues for each lollipop bug. Easier to track/refer to.

hwjump commented 9 years ago

@rovo89 Thank you very much for your work, Greate job！ Could you opening the libxposed_art.so's source code for community. I am very interesting in how to impelments of "invokeOriginalMethodNative" on art.

rovo89 commented 9 years ago

That's still on my list, but I'd like to make sure the code is cleaned up before. Ideally, I would also divide the huge amount of changes into several commits, but I have to see how much work that will be.

rovo89 commented 9 years ago

I finally pushed the remaining part of source code (appprocess and libxposed*.so) to the master branch: https://github.com/rovo89/Xposed/commit/ec56cfe697b127f4d4a01f7f89812ba2dd10b378

I will close this issue now. Issues concerning ART should be raised here: https://github.com/rovo89/android_art