Let's port Xposed to N

[abforce]

@rovo89 We know that you're too busy to implement Xposed for N. No problem, just enjoy your life. But let us to do the job. You've got a high level of knowledge how stuff works on Art, and how Xposed should be implemented. but you've no time to do this.

On the other hand, there's lots of people been waiting for it, and of course they have much free time to do this great job, however they don't have a minimum and required insight on how just Art works and how it should be changed.

So, an idea would be sharing the knowledge by you, implementing it by us. Unfortunately Art doesn't come with a minimum and acceptable documentation, which drastically slows down the research on how stuff works. So, if you consider taking your time to write a brief and techninal writeup on the how, we'll have a good chance to keep the Xposed up to date with the latest release of Android.

What do you think? Do you have time for this or would prefer to still enjoy your life (and possibly playing with your puppies)?

Thanks in advance.

[rovo89]

So, an idea would be sharing the knowledge by you, implementing it by us.

The actual implementation is the smallest part of developing Xposed. Getting the knowledge about how the ART compiler and runtime works and finding ways to integrate Xposed with maximum compatiblity is the most time-consuming and complex part. Books could be written about the small, yet very important details that have to be considered. Sometimes when I understood how a particular implementation works and which tricks they're using, I indeed think I should write something up about it, but that would take me even more time (which I currently don't have).

[roshanravan]

@Revo89 its very good to see that you are still pursuing beloved xposed, thank you for that.

Are you also looking into developing xposed for Android O too? Maybe Like Marshmallow some things even made it possible to integrate Xposed in a more elegant way in Android O and a workaround for JIT & AOT?! (We could only hope for)

We know your time is so little and there are many difficulties but seeing you replying replying gives xposed fan lots of hope

[abforce]

@rovo89 I just got my hands dirty. Please take a look at my copy: https://github.com/abforce/xposed_art_n. Everything works smoothly and flawlessly but XPrivacy. It breaks the boot process. I've posted an issue there along with its logcat output.

I'm heading toward building a built-in enabled Xposed firmware, so I was just porting commits relating to hooking methods not those relating to recompiling. See diff to get a clear insight on what I've done.

[roshanravan]

@abforce very interesting job. If you are using pro version of xprivacy it has a stupid protection which checks for apps like LuckyPatcher and lvl emulation when device is booted. Maybe you break one of his checks during the boot. Can i ask about the firmware you said? It's for what device?

[abforce]

@roshanravan

Can i ask about the firmware you said? It's for what device?

Sure, please ask them on the issue page.

[roshanravan]

@abforce or may call you AliReza Aint you the person who started that project? after reading your project this is my understanding that those modification should be done in N Rom? something like developing a custom Rom? right? in this way there is no need to xposed installer apk? if not how we check for module's update to control on which modules to be enabled/disabled?

i have OnePlust 3T and a Samsung Note4 n910c, unfortunately at the moment i use my OP3T for business purposes so i can not do testings on it , and there is no Android N for my Note4. but when your project gets stable enough and one made those ART modification to the N Rom i will flash it on my OP3T.

[rovo89]

Congrats ;) Unfortunately, I can't review your changes, it's just too much and I don't remember all those details where I deviated from simply porting the Marshmallow changes and which additional changes were required to make it reliable (i.e. to ensure that the hooks are called even with compiler optimizations).

As you already mentioned, your changes could only work for ROMs compiled from scratch with Xposed already active, so people should understand that this won't work (at least not reliably) on their stock or otherwise pre-compiled ROMs.

[abforce]

@rovo89 Thanks for your prompt feedbacks. Surprisingly by porting M changes to N everything works well, EnableXposedHook, redirecting calls to proxy handler and then going back to Java part and calling callbacks. I believe that hooking functionality works possibly well. But there's a problem when GC runs, specifically I suspect that heap or stack will be malformed when XPrivacy is enabled.

As I traced down the error, the problem seems to be related to visiting roots when GC marks objects. Moving form M to N, just one of your commits I couldn't port directly, that of ArtMethod::VisitRoots.

Also enabling hook resources causes the problem when GC runs, are you sure your modified Object.Clone() doesn't cause memory issues?

I can't review your changes, it's just to

No, that's not too much. Please only review those parts relating to visiting roots. I'll heartily appreciate your effort. There's lots of people waiting for my custom ROM.

Just a moment please stop playing with your puppies and give them a rest. They really do need it. Just joking, don't take it personally :laughing:

[dragonpt]

Congrats, this indeed seems just great Kinda surprised, the M changes do work.

Either way i may give a look at building aosp with "patched" art

[roshanravan]

Thanks to @abforce and BlackSoulxxx There is a flashable xposed for N available here: https://forum.xda-developers.com/xposed/unofficial-xposed-v87-unofficial-yes-im-t3639192 It seems like it's for AOSP ROMs Good job, we know Revo89 also working on different release from scratch but it will be so good if he helps and guide. As he is grand master.

[dragonpt]

haha yup :)

[newstart]

https://forum.xda-developers.com/xposed/xposed-android-nougat-sdk-25-arm64-t3639221

Sorry Revo, they beat you to the punch.

[wanam]

Be polite people.

[Ninlives]

@newstart Don't they use rovo's codes? It can't exist without his work on Marshmallow, I don't think that's a punch.

[newstart]

Here are more builds, more stable.... surprisingly.

https://drive.google.com/drive/folders/0B5ePoKNP9UFtWGRVN2JlazhYU2c

[aviraxp]

@newstart They use @abforce's source entirely. The "mod" they made is just about building.

[dragonpt]

Exactly

[rovo89]

Surprisingly by porting M changes to N everything works well,

From my experience, I can tell you that it's indeed not that hard to get the basics done, but it takes a hell lot of time for the details. It's like the 80-20 rule, except that it's more like 95-5 or so.

The hooking approach I have designed for the previous releases is rather stable and can be applied for newer releases as well. I assume you needed most of the time to check all the places where IsProxyMethod() and IsDirect() is called and decide whether they should consider the new method modifier or not (i.e. use parameter true or not). If you didn't do that, you might have missed some places where they have newly introduced such calls. Same for methods like InstallStubsForMethod(), which needs to operate on the original original method even when they're called for a hooked method. You'd need to check for more places where this could be relevant. That's still in the range of days, but if you miss to do it, you might see obscure crashes or misbehavior that are very hard to debug.

Besides that, you took an approach where hooks would only work properly for ROMs which are entirely compiled from scratch. On stock ROMs and even most custom ROMs (that didn't include your port while building) as well as for any apps the device might have compiled before. So you'd have to start all over again with a wiped device running a special ROM. If not, then optimizations (e.g. inlining) might prevent a method from being called at all, and hence hooks wouldn't work. The worst thing about this is that the exact behavior would depend on the ROM, or even how the user uses the device (now that apps are compiled based on profiles). Modules would fail randomly and the module developers would be wondering what they did wrong (answer: nothing). If not all of the prerequisites are fulfilled, it's a pretty unreliable API. Imagine Google published an update where the OnClickListener of a button wasn't called sometimes, unless you compiled your ROM from scratch... For me personally, this is a big limitation.

There is a flashable xposed for N available here: https://forum.xda-developers.com/xposed/unofficial-xposed-v87-unofficial-yes-im-t3639192 It seems like it's for AOSP ROMs

Their code is @abforce's port, with all the limitations I listed above. 🙄 It doesn't make sense at all to publish flashable ZIPs with this, as the port is only intended for recompiling a ROM from scratch. And as I feared, there is not a single warning that this is an incomplete port that will often fail to fulfill the API contract. 😠 That's bad for developers and users, they should at least know it's incomplete so they can decide whether they want to support/use it. I would never dare to do something like that without a big fat warning. Actually, as I'm rather perfectionist, I wouldn't publish an incomplete version at all, reliablity is very important for me.

I just checked my Git log and I had the basic hooking (as in this port) ready in October. Since then, I have worked on removing the limitations. To make it more reliable on pre-compiled ROMs, I could have ported the recompiling part as well, however that's still not ideal. As explained somewhere above, we have JIT now, which allows us to keep all the optimizations, only invalidating (and possibly recompiling) the methods which are directly affected by hooking. This is quite challenging, as it's more than just copy & paste. And until this isn't done, I won't publish anything. 9 months for that (until now) is really a long time, I know, but hey, it's my spare time and e.g. due to moving to a new apartment, I couldn't work on Xposed for several months (!) at all, and had only very limited time otherwise.

As I traced down the error, the problem seems to be related to visiting roots when GC marks objects.

Also enabling hook resources causes the problem when GC runs, are you sure your modified Object.Clone() doesn't cause memory issues?

I just commented on your diff. There's no real difference in my code for these two methods, so you'll have to debug it yourself. I know this can be hard, as the real culprit can sometimes be in a totally different place. Even if it ends up to be a single character that needs to be changed, you can easily spend dozens of hours on debugging one issue. That said - if you're seriously interested in contributing, this might be a good challenge. 😉

[aviraxp]

Totally agree with rovo, from my test, my module (which hooks many fundamental classes like android.view.ViewGroup) would fail randomly with many native crashes from the hooked app. Although we can install it (even flash it) with a not pre-odexed rom and cleaning every partition to clear odex files, it is not stable at all and cannot be used for production environment.

[dragonpt]

Rovo of course Im totally with you on this.

Also prebuilt stuff its not the way, I'll be doing (even with all the limitations ) patch our ROM source, that's really the only way to at least debug and not just flash an miracle prebuilt stuff.zip

We all appreciate your work rovo, and I know how long this stuff can takes when done on our limited free Tim Cheers

[rovo89]

And just to add to my reply above: Imagine I published my current WIP now. I bet that some people would publish it as their build, without any warnings that it's incomplete and not intended to be used productively.

[dragonpt]

Surely !

[randomhydrosol]

Can confirm this. I build with full dexpreopt and dexpreopt_pic and the unofficial xposed causes multiple soft-reboots repeatedly even when modules are not enabled on my ROM

[dragonpt]

Just saw xda's "Purify" and there's an higher BS there.. For real... Asking donations in your behalf using ABForce work patch up & ofc the whole rovo's work

[Spektykles]

And there is an XDA News article backed it up...

[dragonpt]

Yup top notch ... /irony

[mo3rfan]

Slightly offtopic, but I was also interested in hacking around to get Xposed to work on Nougat. The only thing that put me off is the huge resource requirements for the dev environment. 100 GB+ of disk space/bandwidth and large RAM requirements just to get AOSP to compile! Is there a better way to work with android/ART? How do you guys do it? @rovo89 @abforce

[dragonpt]

Its the normal environment to get aosp compiled .. You could get away with just 8gb Ram a fairly CPU and ofc the HDD space (ssd even better) if you don't have those requirements, maybe search for a cloud/build server solution

[rovo89]

Yep, that quite unfortunate, but that's because AOSP source is so huge. The only hint I can share is to initialize like this:

repo init -u https://android.googlesource.com/platform/manifest -g default,-device,-darwin,-cts,platform-linux -b <branch>

With the -g parameter, you can limit the repos that should be downloaded to certain groups. Excluding some groups can avoid to download stuff that's not needed for ART anyway (but much other stuff is actually required).

Besides that, if you're planning to check out multiple versions of AOSP (e.g. Lollipop, Marshmallow and Nougat), it helps to symlink the ".repo/project-objects" folder to a shared place before fetching the source code.

[Vynikal]

So... if the Xposed code is compiled with the ROM, is it stable without any said limitations? If yes, it would be nice to squash the changes to one commit to be applied to source tree, so ROM developers could use it. Just a suggestion. EDIT: As I see, it would mean change the ART repository according to abforce's version and replacing app_process package, am I right?

[randomhydrosol]

It is possible You need to write a makefile for xposedbridge and have it compile during compilation Then pick rovo's changes to art, frameworks and others

[randomhydrosol]

The main issue is the app process binary Why? Because "replacing" the stuff in frameworks/base seriously won't give you stable results as the app_process source with xposed seems to be based on very old sources. If you think that "replacing" the app_process in frameworks is going to work without errors then you just don't know what you're doing

[dragonpt]

Just looking at "replace" base stuff , gives me headaches lol ...that's dirty... Surely you'll have some awesome build errors ;) haha

[Vynikal]

I knooow, folks, I'm noob at this. It was more like a suggestion, it would be nice to get a stable Xposed either way.

[aviraxp]

No matter how Xposed is compiled, it is not stable. The issue is in the code itself, rather than building.

[dragonpt]

What @aviraxp says...

[wanam]

@rovo89 I compared the Java part of Samsung Art (Android 7.0) to the aosp sources, and it seems Samsung do not use the extra fields and native methods we used to implement here and here , the only remaining one i'm seeing is "String.clear()".

To avoid Knox troubles i guess we will still need to hook these methods on the bridge: com.samsung.android.security.CCManager.isMdfEnforced() and a new one com.android.org.conscrypt.NativeCrypto.isMdfEnforced()

If you are willing to try your WIP work against Samsung 7.0 Roms for eventual official support, i will be happy to help.

[dragonpt]

@wanam well that's an surprise ... Great find

[svg-frog]

@rovo89 So, if I was so inclined, I could download the entirety of AOSP, build the patch from @abforce into it, and compile (RIP using my computer for a while), then eventually, should that be loaded onto my phone (OnePlus 3), it should work fine? Or would there still be the random failures/crashes issue?

[randomhydrosol]

That name

[dragonpt]

C'mon you should know... Don't ask ETA's !

[dragonpt]

Also...with the amount of "releases" now I've seen on XDA (and on other places ) ...those dirty prebuilt releases... I wouldn't be happy ...

People forget credits and aren't ashamed of that... Newbs

[wanam]

Surprisingly it booted with my Samsung GS7 running stock FW;

07-21 10:39:28.912 3288 3288 I Xposed : ----------------- 07-21 10:39:28.912 3288 3288 I Xposed : Starting Xposed version 87.1 (custom build by Wanam / 20170721), compiled for SDK 24 07-21 10:39:28.912 3288 3288 I Xposed : Device: SM-G930F (samsung), Android version 7.0 (SDK 24) 07-21 10:39:28.912 3288 3288 I Xposed : ROM: NRD90M.G930FXXU1DQEF 07-21 10:39:28.912 3288 3288 I Xposed : Build fingerprint: samsung/heroltexx/herolte:7.0/NRD90M/G930FXXU1DQEF:user/release-keys 07-21 10:39:28.912 3288 3288 I Xposed : Platform: arm64-v8a, 64-bit binary, system server: yes 07-21 10:39:28.912 3288 3288 I Xposed : SELinux enabled: yes, enforcing: yes 07-21 10:39:39.040 3289 3289 I Xposed : ----------------- 07-21 10:39:39.040 3289 3289 I Xposed : Starting Xposed version 87.1 (custom build by Wanam / 20170721), compiled for SDK 24 07-21 10:39:39.040 3289 3289 I Xposed : Device: SM-G930F (samsung), Android version 7.0 (SDK 24) 07-21 10:39:39.040 3289 3289 I Xposed : ROM: NRD90M.G930FXXU1DQEF 07-21 10:39:39.040 3289 3289 I Xposed : Build fingerprint: samsung/heroltexx/herolte:7.0/NRD90M/G930FXXU1DQEF:user/release-keys 07-21 10:39:39.040 3289 3289 I Xposed : Platform: arm64-v8a, 32-bit binary, system server: no 07-21 10:39:39.040 3289 3289 I Xposed : SELinux enabled: yes, enforcing: yes 07-21 10:39:39.043 3289 3289 I Xposed : ----------------- 07-21 10:39:39.043 3289 3289 I Xposed : Added Xposed (/system/framework/XposedBridge.jar) to CLASSPATH 07-21 10:39:30.859 3288 3288 I Xposed : Loading modules from /data/app/ma.wanam.twitteradfree-1/base.apk 07-21 10:39:30.866 3288 3288 I Xposed : Loading class ma.wanam.twitteradfree.XTwitter 07-21 10:39:30.867 3288 3288 I Xposed : Loading modules from /data/app/ma.wanam.xbein-1/base.apk 07-21 10:39:30.872 3288 3288 I Xposed : Loading class ma.wanam.xbein.BeIN 07-21 10:39:30.873 3288 3288 I Xposed : Loading modules from /data/app/ma.wanam.youtubeadaway-1/base.apk 07-21 10:39:30.879 3288 3288 I Xposed : Loading class ma.wanam.youtubeadaway.Xposed

I tried with my 3 simple modules, and the hooks seems to work.

Resources hooking is broken because somehow "xposed_XResourcesSuperClass.dex" is created with 600 permissions.

07-21 09:59:17.229 3680 3680 E dex2oat : Failed to read magic number from dex file: '/data/dalvik-cache/xposed_XResourcesSuperClass.dex': Permission denied

This can be fixed manually by setting its perms to 644.

07-21 10:08:36.630 3766 3766 I dex2oat : /system/bin/dex2oat --dex-file=/data/dalvik-cache/xposed_XResourcesSuperClass.dex --oat-fd=14 --oat-location=/data/dalvik-cache/arm64/data@dalvik-cache@xposed_XResourcesSuperClass.dex --compiler-filter=speed 07-21 10:08:36.722 3766 3766 I dex2oat : dex2oat took 96.503ms (threads: 8) arena alloc=0B (0B) java alloc=42KB (43200B) native alloc=942KB (965216B) free=1617KB (1656224B)

I got also the same silent crash of "VisitRoots" on many system apps, to summarize this port is not ready for end user, and i will NOT publish any builds.

I'm happy that Samsung didn't make it harder, let's wait for @rovo89's release.

[dragonpt]

@wanam that's great

I assume, NativeCrypto, wasn't needed ?

[wanam]

@dragonpt According to Samsung's conscrypt sources, it is needed to support 2 ssl crypto algos.

[Raoao]

If you can write a Magisk module, is the same for the xposed framework can hijack api.Magisk framework and xposed framework principle, I think this is feasible, perhaps is a perfect solution, no longer need to compile the source code, the use of more simple (Note: I do not speak English, the content of translation by Google)

[svg-frog]

@Raoao Someone did make a Magisk module out of it - Just as unstable as flashing it over a already existing ROM.

[Ninlives]

@Raoao Magisk modules are not magical.… It will not actually modify your /system folder but in other aspects there is no difference between using an Magisk module or actually flashing an unstable, WIP zip.

[WilliamO7]

@Ninlives But what if you built a ROM and installed that, and then created a magisk module out of those files?

[NHellFire]

And just to add to my reply above: Imagine I published my current WIP now. I bet that some people would publish it as their build, without any warnings that it's incomplete and not intended to be used productively. @rovo89 if you share it, we might get some more progress on it. Since it sounds like @abforce and @wanam would look at it. Other people may have different ideas to try and more time to work on it.