Cardboard SDK sensor API bypass dispatchSensorEvent

[lrq3000]

So it seems that recently, an update to the Google Cardboard SDK introduced its own sensor API with automatic drift correction and stuff like that. For VR apps, it's now the advised way to access the sensors, instead of the android.hardware.Sensor API.

The bug was first spotted with Cmoar Cinema Pro (not the Free, it doesn't use the Cardboard API) but it can easily be reproduced using VRTV Video Player Free: this app provides an option "Orientation Provider" with 4 choices:

1. Google Cardboard SDK (default): this produces the aforementioned bug.
2. Manual touch input: use the touch screen to move the screen instead of the sensors. It also bypasses our module but that's normal.
3. Improved Orientation Sensor: no problem, the module works.
4. Calibrated Gyroscope: no problem also.

It seems this Cardboard sensor API is quite new so I don't have much more info.

[lrq3000]

What do you think about it Kjos? I'm not really used to hacking into sourcecode using Xposed. If I find the documentation, do you think you can hack some quick code? (I can try it out and enhance it, I just don't know how to hook properly).

[Kjos]

My thought on this, is that the new Cardboard SDK uses the NDK to get the Gyro readings. See: https://source.android.com/devices/halref/sensors_8h_source.html and other source files. I'm not very familiar with using the NDK, but I'm pretty sure accessing the Gyro through the NDK omits and Java calls. Xposed only reads the Java calls, so there's no way to access those. A quick fix would be to look into the Cardboard jar and see what functions are exposed. If any of them accesses the Gyro data, you can hook into that. This used to be the case, as I did that earlier, but not sure with the new SDK, chances are slim tbh. If you're unfamiliar with hooking, I can recommend reading the tutorial here: http://forum.xda-developers.com/showthread.php?t=2709324 If you're hooking in Android it helps to have the source code available (and take any changes between Android versions in to mind). I'll take a look at the Cardboard sdk and report what I find.

[lrq3000]

Ok thank you very much for the infos and for looking at this issue, I am also reading the Cardboard sourcecode to try to find somewhere to hook.

2015-09-17 13:20 GMT+02:00 Kjos notifications@github.com:

My thought on this, is that the new Cardboard SDK uses the NDK to get the Gyro readings. See: https://source.android.com/devices/halref/sensors_8h_source.html and other source files. I'm not very familiar with using the NDK, but I'm pretty sure accessing the Gyro through the NDK omits and Java calls. Xposed only reads the Java calls, so there's no way to access those. A quick fix would be to look into the Cardboard jar and see what functions are exposed. If any of them accesses the Gyro data, you can hook into that. This used to be the case, as I did that earlier, but not sure with the new SDK, chances are slim tbh. If you're unfamiliar with hooking, I can recommend reading the tutorial here: http://forum.xda-developers.com/showthread.php?t=2709324 If you're hooking in Android it helps to have the source code available (and take any changes between Android versions in to mind). I'll take a look at the Cardboard sdk and report what I find.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141047645 .

[Kjos]

I think this function needs hooking: private native void nativeGetCurrentEyeParams(long l, com.google.vrtoolkit.cardboard.HeadTransform headTransform, com.google.vrtoolkit.cardboard.Eye eye, com.google.vrtoolkit.cardboard.Eye eye1, com.google.vrtoolkit.cardboard.Eye eye2, com.google.vrtoolkit.cardboard.Eye eye3, com.google.vrtoolkit.cardboard.Eye eye4);

in

package com.google.vrtoolkit.cardboard;

class: CardboardViewNativeImpl

Looking at the source, it looks like Cardboard is also applying some lowpass filtering. However it's no use to mess with that, as I'm pretty sure the native impl doesn't use it (it's in compiled code).

Add to first thing: Maybe better to hook into public void getCurrentEyeParams(com.google.vrtoolkit.cardboard.HeadTransform head, com.google.vrtoolkit.cardboard.Eye leftEye, com.google.vrtoolkit.cardboard.Eye rightEye, com.google.vrtoolkit.cardboard.Eye monocular, com.google.vrtoolkit.cardboard.Eye leftEyeNoDistortionCorrection, com.google.vrtoolkit.cardboard.Eye rightEyeNoDistortionCorrection) { /* compiled code */ }

package: com.google.vrtoolkit.cardboard.CardboardViewApi

Although then you would hook into native ndk as well original Java code (so if not take into account, it could hook twice for the same data with Java Cardboard SDK apps). On the other hand that would be more future proof, as the native function is private and might change.

[Kjos]

Unfortunately as it's all compiled code, you can't hook directly to the gyro, so you need to filter the results instead.

[lrq3000]

Yes it was to be expected that there would be no direct access to the gyroscope since the authors of the Google Cardboard project wants to provide fusionned/virtual sensors to all Cardboard apps to avoid drifts and miscalibration issues.

Well, I think that hooking to the results of the virtual sensors will be as good as we can get. I'm reading the documentation and I'll try to do that :)

[lrq3000]

Just a question: do you have any access to the source code of the Cardboard SDK? I can't find the source code for the latest version...

[lrq3000]

Found another possible hook:

https://github.com/rsanchezsaez/cardboard-java/blob/master/library/src/main/java/com/google/vrtoolkit/cardboard/sensors/HeadTracker.java#L58

What do you think about it?

[Kjos]

I used the latest Cardboard jar from the Github and Intellij automagically decompiles it except for the native code. You can use any other decompiler probably. It's not obfuscated code either, all the names are intact. You can hook in the SensorEvent but it's not used with the NDK code. You need to hook in the results, as posted above.

[lrq3000]

Ah ok I see, then I'm currently trying to hook that, I'll keep you updated :)

[Kjos]

Here's the class decompiled: http://pastebin.com/1YkXXXca

[lrq3000]

Thank's :)

[Kjos]

http://pastebin.com/xpbabKid This is HeadTransform. I'm not sure if it's better to use reflection or anything, but in theory you could include the Cardboard.jar or only needed source files such as HeadTransform.java to get the headview float array, which holds the rotation. Apply a filter to that, and it will work nicely.

[Kjos]

Ah, you could just use reflection like it is done here as well: https://github.com/Kjos/XposedJitteryGyroFix/blob/master/GyroscopeNoiseFilter/src/net/kajos/gyronoisefilter/GyroscopeNoiseFilter.java#L171 I forgot it was done like that. Just use a similar way to get the float[] array of headview.

[lrq3000]

Ok I agree about HeadTransform, let's start with that. However, what method should I hook? I mean, if I hook getHeadView(), this probably won't always work since other methods such as getQuaternion directly access the property this.headView...

2015-09-17 15:31 GMT+02:00 Kjos notifications@github.com:

http://pastebin.com/xpbabKid This is HeadTransform. I'm not sure if it's better to use reflection or anything, but in theory you could include the Cardboard.jar or only needed source files such as HeadTransform.java to get the headview float array, which holds the rotation. Apply a filter to that, and it will work nicely.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141086144 .

[Kjos]

private native void nativeGetCurrentEyeParams(long l, com.google.vrtoolkit.cardboard.HeadTransform headTransform, com.google.vrtoolkit.cardboard.Eye eye, com.google.vrtoolkit.cardboard.Eye eye1, com.google.vrtoolkit.cardboard.Eye eye2, com.google.vrtoolkit.cardboard.Eye eye3, com.google.vrtoolkit.cardboard.Eye eye4);

in

package com.google.vrtoolkit.cardboard;

class: CardboardViewNativeImpl

That would be the easiest way. If they change the SDK again, it may need changing again.

---

Alternatively: public void getCurrentEyeParams(com.google.vrtoolkit.cardboard.HeadTransform head, com.google.vrtoolkit.cardboard.Eye leftEye, com.google.vrtoolkit.cardboard.Eye rightEye, com.google.vrtoolkit.cardboard.Eye monocular, com.google.vrtoolkit.cardboard.Eye leftEyeNoDistortionCorrection, com.google.vrtoolkit.cardboard.Eye rightEyeNoDistortionCorrection) { /* compiled code */ }

package: com.google.vrtoolkit.cardboard.CardboardViewApi

Which has less chance of changing in the future, but you need to make sure no double filtering is applied in older Cardboard apps (it is applied in dispatchSensorEvent (system level) and then again in getCurrentEyeParams (cardboard level)). I'm not sure yet what the most stable way to do that is yet. A well set boolean might just work fine.

[lrq3000]

Ok so you advise me to continue on CardboardViewNativeImpl? Ok then I'll try that, we'll see after about the double filtering issue, I can test that since there are lots of old cardboard apps.

2015-09-17 15:42 GMT+02:00 Kjos notifications@github.com:

private native void nativeGetCurrentEyeParams(long l, com.google.vrtoolkit.cardboard.HeadTransform headTransform, com.google.vrtoolkit.cardboard.Eye eye, com.google.vrtoolkit.cardboard.Eye eye1, com.google.vrtoolkit.cardboard.Eye eye2, com.google.vrtoolkit.cardboard.Eye eye3, com.google.vrtoolkit.cardboard.Eye eye4);

in

package com.google.vrtoolkit.cardboard;

class: CardboardViewNativeImpl

That would be the easiest way. If they change the SDK again, it may need changing again. Alternatively: public void getCurrentEyeParams(com.google.vrtoolkit.cardboard.HeadTransform head, com.google.vrtoolkit.cardboard.Eye leftEye, com.google.vrtoolkit.cardboard.Eye rightEye, com.google.vrtoolkit.cardboard.Eye monocular, com.google.vrtoolkit.cardboard.Eye leftEyeNoDistortionCorrection, com.google.vrtoolkit.cardboard.Eye rightEyeNoDistortionCorrection) { / compiled code / }

package: com.google.vrtoolkit.cardboard.CardboardViewApi

Which has less chance of changing in the future, but you need to make sure no double filtering is applied in older Cardboard apps (it is applied in dispatchSensorEvent (system level) and then again in getCurrentEyeParams (cardboard level)). I'm not sure yet what the most stable way to do that is yet. A well set boolean might just work fine.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141090597 .

[Kjos]

Yeah, better to start with the NativeImpl probably. That won't give any double filtering issues for sure.

[lrq3000]

I could not hook, maybe I did a mistake (since I'm new to Xposed modding), here's the weird error I've got:

MedianFilter for Cardboard API crashed! Exception:

java.lang.classNotFoundException: com.google.vrtoolkit.cardboard.CardboardViewNativeImpl$CardboardViewNativeImpl

Does this mean that we can't hook a private method?

2015-09-17 15:58 GMT+02:00 Kjos notifications@github.com:

Yeah, better to start with the NativeImpl probably. That won't give any double filtering issues for sure.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141095002 .

[lrq3000]

PS: Here's the code I've used. Nothing fancy, I just tried to hook and print a debug message.

http://pastebin.com/rHP5im2x

Interesting stuff starts at line 191 after the comment // Cardboard NDK API Hook

2015-09-17 16:35 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

I could not hook, maybe I did a mistake (since I'm new to Xposed modding), here's the weird error I've got:

MedianFilter for Cardboard API crashed! Exception:

java.lang.classNotFoundException: com.google.vrtoolkit.cardboard.CardboardViewNativeImpl$CardboardViewNativeImpl

Does this mean that we can't hook a private method?

2015-09-17 15:58 GMT+02:00 Kjos notifications@github.com:

Yeah, better to start with the NativeImpl probably. That won't give any double filtering issues for sure.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141095002 .

[Kjos]

com.google.vrtoolkit.cardboard.CardboardViewNativeImpl$CardboardViewNativeImpl I think this should be: com.google.vrtoolkit.cardboard.CardboardViewNativeImpl But I will look into it.

[Kjos]

Yes, pretty sure it should be like that. https://github.com/rovo89/XposedBridge/wiki/Development-tutorial

[Kjos]

Woops, pressed the wrong button.

[lrq3000]

Ok then, I'll leave it up to you. I tried a few other ways to hook, but they all failed :/

2015-09-17 17:51 GMT+02:00 Kjos notifications@github.com:

Woops, pressed the wrong button.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141129782 .

[Kjos]

I'll build a test version tomorrow. I'm hoping you're willing to run it, as I don't have KitKat anymore. My guess is that there needs to be filtered on package, like so:

if (!lpparam.packageName.equals("com.google.vrtoolkit.cardboard")) return;

Before the second findClass call.

Else it will try to hook in libraries that don't have the hookfunction. That would explain java.lang.classNotFoundException . To verify it's a smart move to log the pparam.packageName.

[lrq3000]

Of course I'll gladly try your changes :) I'll be available tomorrow for the tests (but I won't be at all this week-end, I'll be back on Monday).

I tried your idea of a package filter condition, but then the condition is never met, because this package never shows up (since it's only used as a library in other apps, so the package that is displayed to Xposed is the parent app, not Cardboard's).

Also, I can confirm that the exception classNotFoundException also happens inside VR apps that use the Cardboard NDK such as VRTV, and the hook just never works (I have a log debug statement that should be triggered when the hook is correctly done, but it never happens).

Maybe I did something wrong in my code so you should probably retry your ideas on your end to make sure.

2015-09-17 20:44 GMT+02:00 Kjos notifications@github.com:

I'll build a test version tomorrow. I'm hoping you're willing to run it, as I don't have KitKat anymore. My guess is that there needs to be filtered on package, like so:

if (!lpparam.packageName.equals("com.google.vrtoolkit.cardboard")) return;

Before the second findClass call.

Else it will try to hook in libraries that don't have the hookfunction. That would explain java.lang.classNotFoundException . To verify it's a smart move to log the pparam.packageName.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141183954 .

[Kjos]

Can you try the code I added in the "cardboard"-branch? I tried to install Cyanogenmod, but it uses Lollipop and there is no KitKat version available for Moto G 2014. Could you paste the relevant log info too (tag: GyroFilter)?

[lrq3000]

Trying it right away :)

2015-09-18 20:35 GMT+02:00 Kjos notifications@github.com:

Can you try the code I added in the "cardboard"-branch? I tried to install Cyanogenmod, but it uses Lollipop and there is no KitKat version available for Moto G 2014. Could you paste the relevant log info too (tag: GyroFilter)?

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141529965 .

[lrq3000]

Just tried your changes, sorry but there's nothing at all, no debug output relative to the new hooks (I changed the previous debug statements so that they now also use the TAG variable, to make sure I was correctly receiving the logcat and that the filter was correct, those debug statements print correctly as before).

So it seems that the hooks are not even called...

/EDIT: I added a debug print in the catch block for each cardboard hook, and both produce a ClassNotFoundException...

2015-09-18 20:36 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

Trying it right away :)

2015-09-18 20:35 GMT+02:00 Kjos notifications@github.com:

Can you try the code I added in the "cardboard"-branch? I tried to install Cyanogenmod, but it uses Lollipop and there is no KitKat version available for Moto G 2014. Could you paste the relevant log info too (tag: GyroFilter)?

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141529965 .

[Kjos]

Is it possible to chat on IRC? I'm on freenode #xposedgyrofix if you can. I think I've had this problem before. I started solving it by printing all the package names, so that's a good start, if you haven't already. Use an if statement to filter out all the packages that don't matter then, and watch for the exceptions that matter when this filter is applied.

[lrq3000]

Yes I'm connecting right now, sorry for the delay.

2015-09-18 21:05 GMT+02:00 Kjos notifications@github.com:

Is it possible to chat on IRC? I'm on freenode #xposedgyrofix if you can. I think I've had this problem before. I started solving it by printing all the package names, so that's a good start, if you haven't already. Use an if statement to filter out all the packages that don't matter then, and watch for the exceptions that matter when this filter is applied.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-141537712 .

[lrq3000]

Hello Kjos,

To summary, I tried several things but I cannot find a way to make ant-jittering work with Cardboard SDK apps. However, I have progressed a bit. Now for the details...

The code you've made correctly hooks to the HeadTransform.getHeadView() method ("Hook1" displayed in logcat), but every subsequent commands were not executed. I have found a way around that, so I can print in logcat debug the result of getHeadView() correctly, and I nullify these values (if my code is correct, I cannot check that since I don't have the sourcecode of the apps I'm testing, but I know the whole code is executing correctly because I have a debug message just after and it's printed correctly). The code can be found here (in my cardboard branch):

https://github.com/lrq3000/XposedJitteryGyroFix/commit/5282e34d3538e2b200299f2db7e9c516c18169a2#diff-eb3a59265af5a1582172a1322805fa14R201

However, it seems it's not enough to tamper with the Cardboard's management of the gyroscope. So either we also have to hook into Eye.getEyeView() (I tried but I couldn't hook), or we need another hook (I tried the other ones you cited in your previous messages, but then I get the old errors that the classes cannot be found - maybe this is because of a different cardboard version? It seems that the cardboard's classes are changing drastically, and the classes you mentioned are only available in the very latest SDK version, so if the apps are using an older version they are not available...).

So, I think we are stuck because: 1- we don't have the correct hook yet. We'd need a developer who is experienced with developping apps for the Cardboard. 2- the cardboard SDK is changing so fast that classes are often changing. This really doesn't make this task easy on us.

So, what I tried to do, is that I tried to run the TreasureHunt example cardboard project, which can be found compiled here: https://play.google.com/store/apps/details?id=com.ccc.vrtoolkit.cardboard.samples.treasurehunt

And indeed, our module does not work on this app, just like VRTV and Cmoar Cinema Pro, so I guess we can use the TreasureHunt project to fix this issue in a more efficient way, because we have the sourcecode of TreasureHunt:

https://github.com/googlesamples/cardboard-java For eclipse: https://github.com/raasun/treasurehunt

However, I cannot find a way to compile it yet on my machine (using Eclipse, the original project is made to be run with Android Studio). I'm going to give it a last try today and see what I can do.

[Kjos]

Hooking is a very delicate process. You can create a successful hook for one application and then it might not work in the other. Hooking in an external library is harder than hooking in the system, so it is indeed a challenging task. I do think it's possible having said that, but it is the question whether it is worth the effort.

Do you have time to get on IRC? I'm there now, same place. If you want to continue working on this, I'm willing to help.

[lrq3000]

The hooks are working correctly, so this is not an issue with hooking, but only about where to hook and what to do with that hook.

Ok for IRC, I'm connecting right now :)

2015-09-23 12:49 GMT+02:00 Kjos notifications@github.com:

Hooking is a very delicate process. You can create a successful hook for one application and then it might not work in the other. Hooking in an external library is harder than hooking in the system, so it is indeed a challenging task. I do think it's possible having said that, but it is the question whether it is worth the effort.

Do you have time to get on IRC? I'm there now, same place. If you want to continue working on this, I'm willing to help.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-142564172 .

[lrq3000]

A last update about this issue: so with your code and some more work, I could make the hook work on Eye.getEyeView(), and this fix the issue for most Cardboard apps.

However, Cmoar Cinema Pro v4.3 still isn't supported. I tried to hook into android.hardware.GeomagneticField to check if maybe the app is not using the gyroscope but another orientation provider, but it doesn't work. In fact, all the hooks we implemented are hooking into Cmoar (debug messages are displayed), which means that Cmoar indeed is using the cardboard SDK, but then the hooks are never called (there's never the "Hookx !" message).

What we know using the decompilation of its classes is that Cmoar seems to be using Unity3d, so maybe this issue is affecting all Unity3d apps.

Or maybe not, maybe it's not directly related to Unity3d but just to a third-party library over Unity3d, such as this one: http://zeissvrone.tumblr.com/post/105593393832/new-unity3d-sdk-11-version-with-integrated-native https://bitbucket.org/vrone/unity3d https://bitbucket.org/vrone/unity3d#markdown-header-how-do-i-use-the-head-tracking-library-natively-without-unity

I tried to hook on this one and it didn't work, so Cmoar is not using this library, but maybe another one of a similar kind is used.

/EDIT: confirmed, it seems that all Unity3D projects suffer from this issue: https://play.google.com/store/apps/details?id=com.nexus.VRscene

I'm stopping here for today... I'll take a look tomorrow, but if I don't find a way I'll pass for the moment, until there's more documentation :/

2015-09-23 15:28 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

The hooks are working correctly, so this is not an issue with hooking, but only about where to hook and what to do with that hook.

Ok for IRC, I'm connecting right now :)

2015-09-23 12:49 GMT+02:00 Kjos notifications@github.com:

Hooking is a very delicate process. You can create a successful hook for one application and then it might not work in the other. Hooking in an external library is harder than hooking in the system, so it is indeed a challenging task. I do think it's possible having said that, but it is the question whether it is worth the effort.

Do you have time to get on IRC? I'm there now, same place. If you want to continue working on this, I'm willing to help.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-142564172 .

[lrq3000]

Found it: it's only Unity3d projects using the Durovis Dive SDK:

https://www.durovis.com/sdk.html

More specifically, there's a lib that is called: /mnt/asec/com.divegames.diveDemo-1/lib/libdivesensor.so. I don't know where to find it, it showed up when my phone asked for permission (due to Xprivacy).

For example, these apps won't work with the GyroNoiseFilter: https://play.google.com/store/apps/details?id=com.divegames.diveDemo https://play.google.com/store/apps/details?id=com.nexus.VRscene

And both are using the Durovis Dive SDK. Cmoar Cinema Pro v4.3 must also probably use this SDK, as it implements the same workarounds for anti-drifting.

On the other hand, there are other libraries for head tracking in Unity for Android, because this is not implemented natively, so developers must buy a head tracking library: http://www.talkingquickly.co.uk/2014/11/google-cardboard-unity-tutorial/

Another such library is: https://www.assetstore.unity3d.com/en/#!/content/25766

With demo app here: https://play.google.com/store/apps/details?id=com.upft.vr.headtracker.demo

But this app is supported by our module, so no problem. So I think the problem is only with Durovis Dive, which somehow bypass the sensor manager.

2015-09-24 2:30 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

A last update about this issue: so with your code and some more work, I could make the hook work on Eye.getEyeView(), and this fix the issue for most Cardboard apps.

However, Cmoar Cinema Pro v4.3 still isn't supported. I tried to hook into android.hardware.GeomagneticField to check if maybe the app is not using the gyroscope but another orientation provider, but it doesn't work. In fact, all the hooks we implemented are hooking into Cmoar (debug messages are displayed), which means that Cmoar indeed is using the cardboard SDK, but then the hooks are never called (there's never the "Hookx !" message).

What we know using the decompilation of its classes is that Cmoar seems to be using Unity3d, so maybe this issue is affecting all Unity3d apps.

Or maybe not, maybe it's not directly related to Unity3d but just to a third-party library over Unity3d, such as this one:

http://zeissvrone.tumblr.com/post/105593393832/new-unity3d-sdk-11-version-with-integrated-native https://bitbucket.org/vrone/unity3d

https://bitbucket.org/vrone/unity3d#markdown-header-how-do-i-use-the-head-tracking-library-natively-without-unity

I tried to hook on this one and it didn't work, so Cmoar is not using this library, but maybe another one of a similar kind is used.

/EDIT: confirmed, it seems that all Unity3D projects suffer from this issue: https://play.google.com/store/apps/details?id=com.nexus.VRscene

I'm stopping here for today... I'll take a look tomorrow, but if I don't find a way I'll pass for the moment, until there's more documentation :/

2015-09-23 15:28 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

The hooks are working correctly, so this is not an issue with hooking, but only about where to hook and what to do with that hook.

Ok for IRC, I'm connecting right now :)

2015-09-23 12:49 GMT+02:00 Kjos notifications@github.com:

Hooking is a very delicate process. You can create a successful hook for one application and then it might not work in the other. Hooking in an external library is harder than hooking in the system, so it is indeed a challenging task. I do think it's possible having said that, but it is the question whether it is worth the effort.

Do you have time to get on IRC? I'm there now, same place. If you want to continue working on this, I'm willing to help.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-142564172 .

[lrq3000]

Might also be of interest: http://stackoverflow.com/questions/30046425/how-can-i-find-the-rotation-of-the-head-in-google-cardboard-unity3d

2015-09-24 3:13 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

Found it: it's only Unity3d projects using the Durovis Dive SDK:

https://www.durovis.com/sdk.html

More specifically, there's a lib that is called: /mnt/asec/com.divegames.diveDemo-1/lib/libdivesensor.so. I don't know where to find it, it showed up when my phone asked for permission (due to Xprivacy).

For example, these apps won't work with the GyroNoiseFilter: https://play.google.com/store/apps/details?id=com.divegames.diveDemo https://play.google.com/store/apps/details?id=com.nexus.VRscene

And both are using the Durovis Dive SDK. Cmoar Cinema Pro v4.3 must also probably use this SDK, as it implements the same workarounds for anti-drifting.

On the other hand, there are other libraries for head tracking in Unity for Android, because this is not implemented natively, so developers must buy a head tracking library: http://www.talkingquickly.co.uk/2014/11/google-cardboard-unity-tutorial/

Another such library is: https://www.assetstore.unity3d.com/en/#!/content/25766

With demo app here: https://play.google.com/store/apps/details?id=com.upft.vr.headtracker.demo

But this app is supported by our module, so no problem. So I think the problem is only with Durovis Dive, which somehow bypass the sensor manager.

2015-09-24 2:30 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

A last update about this issue: so with your code and some more work, I could make the hook work on Eye.getEyeView(), and this fix the issue for most Cardboard apps.

However, Cmoar Cinema Pro v4.3 still isn't supported. I tried to hook into android.hardware.GeomagneticField to check if maybe the app is not using the gyroscope but another orientation provider, but it doesn't work. In fact, all the hooks we implemented are hooking into Cmoar (debug messages are displayed), which means that Cmoar indeed is using the cardboard SDK, but then the hooks are never called (there's never the "Hookx !" message).

What we know using the decompilation of its classes is that Cmoar seems to be using Unity3d, so maybe this issue is affecting all Unity3d apps.

Or maybe not, maybe it's not directly related to Unity3d but just to a third-party library over Unity3d, such as this one:

http://zeissvrone.tumblr.com/post/105593393832/new-unity3d-sdk-11-version-with-integrated-native https://bitbucket.org/vrone/unity3d

https://bitbucket.org/vrone/unity3d#markdown-header-how-do-i-use-the-head-tracking-library-natively-without-unity

I tried to hook on this one and it didn't work, so Cmoar is not using this library, but maybe another one of a similar kind is used.

/EDIT: confirmed, it seems that all Unity3D projects suffer from this issue: https://play.google.com/store/apps/details?id=com.nexus.VRscene

I'm stopping here for today... I'll take a look tomorrow, but if I don't find a way I'll pass for the moment, until there's more documentation :/

2015-09-23 15:28 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

The hooks are working correctly, so this is not an issue with hooking, but only about where to hook and what to do with that hook.

Ok for IRC, I'm connecting right now :)

2015-09-23 12:49 GMT+02:00 Kjos notifications@github.com:

Hooking is a very delicate process. You can create a successful hook for one application and then it might not work in the other. Hooking in an external library is harder than hooking in the system, so it is indeed a challenging task. I do think it's possible having said that, but it is the question whether it is worth the effort.

Do you have time to get on IRC? I'm there now, same place. If you want to continue working on this, I'm willing to help.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-142564172 .

[lrq3000]

And here's another head tracking SDK that is not supported by our module: http://forum.unity3d.com/threads/gyrodroid-access-each-and-every-sensor-on-android-devices-released.101279/

With demo app here: https://play.google.com/store/apps/details?id=com.prefrontalcortex.gyrodroid

2015-09-24 3:18 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

Might also be of interest:

http://stackoverflow.com/questions/30046425/how-can-i-find-the-rotation-of-the-head-in-google-cardboard-unity3d

2015-09-24 3:13 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

Found it: it's only Unity3d projects using the Durovis Dive SDK:

https://www.durovis.com/sdk.html

More specifically, there's a lib that is called: /mnt/asec/com.divegames.diveDemo-1/lib/libdivesensor.so. I don't know where to find it, it showed up when my phone asked for permission (due to Xprivacy).

For example, these apps won't work with the GyroNoiseFilter: https://play.google.com/store/apps/details?id=com.divegames.diveDemo https://play.google.com/store/apps/details?id=com.nexus.VRscene

And both are using the Durovis Dive SDK. Cmoar Cinema Pro v4.3 must also probably use this SDK, as it implements the same workarounds for anti-drifting.

On the other hand, there are other libraries for head tracking in Unity for Android, because this is not implemented natively, so developers must buy a head tracking library: http://www.talkingquickly.co.uk/2014/11/google-cardboard-unity-tutorial/

Another such library is: https://www.assetstore.unity3d.com/en/#!/content/25766

With demo app here: https://play.google.com/store/apps/details?id=com.upft.vr.headtracker.demo

But this app is supported by our module, so no problem. So I think the problem is only with Durovis Dive, which somehow bypass the sensor manager.

2015-09-24 2:30 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

A last update about this issue: so with your code and some more work, I could make the hook work on Eye.getEyeView(), and this fix the issue for most Cardboard apps.

However, Cmoar Cinema Pro v4.3 still isn't supported. I tried to hook into android.hardware.GeomagneticField to check if maybe the app is not using the gyroscope but another orientation provider, but it doesn't work. In fact, all the hooks we implemented are hooking into Cmoar (debug messages are displayed), which means that Cmoar indeed is using the cardboard SDK, but then the hooks are never called (there's never the "Hookx !" message).

What we know using the decompilation of its classes is that Cmoar seems to be using Unity3d, so maybe this issue is affecting all Unity3d apps.

Or maybe not, maybe it's not directly related to Unity3d but just to a third-party library over Unity3d, such as this one:

http://zeissvrone.tumblr.com/post/105593393832/new-unity3d-sdk-11-version-with-integrated-native https://bitbucket.org/vrone/unity3d

https://bitbucket.org/vrone/unity3d#markdown-header-how-do-i-use-the-head-tracking-library-natively-without-unity

I tried to hook on this one and it didn't work, so Cmoar is not using this library, but maybe another one of a similar kind is used.

/EDIT: confirmed, it seems that all Unity3D projects suffer from this issue: https://play.google.com/store/apps/details?id=com.nexus.VRscene

I'm stopping here for today... I'll take a look tomorrow, but if I don't find a way I'll pass for the moment, until there's more documentation :/

2015-09-23 15:28 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

The hooks are working correctly, so this is not an issue with hooking, but only about where to hook and what to do with that hook.

Ok for IRC, I'm connecting right now :)

2015-09-23 12:49 GMT+02:00 Kjos notifications@github.com:

Hooking is a very delicate process. You can create a successful hook for one application and then it might not work in the other. Hooking in an external library is harder than hooking in the system, so it is indeed a challenging task. I do think it's possible having said that, but it is the question whether it is worth the effort.

Do you have time to get on IRC? I'm there now, same place. If you want to continue working on this, I'm willing to help.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-142564172 .

[lrq3000]

This last one is more interesting than the Durovis Dive SDK, because here we get to see a new class named "com.pfc.sensors.SensorClass.class". Here's the source code:

http://pastebin.com/N4eYM3P0

It's basically calling the SensorManager and SensorEvent. It's weird that it isn't caught by our hook on SystemSensorManager. What do you think? Is it possible to catch this one?

2015-09-24 3:35 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

And here's another head tracking SDK that is not supported by our module:

http://forum.unity3d.com/threads/gyrodroid-access-each-and-every-sensor-on-android-devices-released.101279/

With demo app here:

https://play.google.com/store/apps/details?id=com.prefrontalcortex.gyrodroid

2015-09-24 3:18 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

Might also be of interest:

http://stackoverflow.com/questions/30046425/how-can-i-find-the-rotation-of-the-head-in-google-cardboard-unity3d

2015-09-24 3:13 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

Found it: it's only Unity3d projects using the Durovis Dive SDK:

https://www.durovis.com/sdk.html

More specifically, there's a lib that is called: /mnt/asec/com.divegames.diveDemo-1/lib/libdivesensor.so. I don't know where to find it, it showed up when my phone asked for permission (due to Xprivacy).

For example, these apps won't work with the GyroNoiseFilter: https://play.google.com/store/apps/details?id=com.divegames.diveDemo https://play.google.com/store/apps/details?id=com.nexus.VRscene

And both are using the Durovis Dive SDK. Cmoar Cinema Pro v4.3 must also probably use this SDK, as it implements the same workarounds for anti-drifting.

On the other hand, there are other libraries for head tracking in Unity for Android, because this is not implemented natively, so developers must buy a head tracking library: http://www.talkingquickly.co.uk/2014/11/google-cardboard-unity-tutorial/

Another such library is: https://www.assetstore.unity3d.com/en/#!/content/25766

With demo app here:

https://play.google.com/store/apps/details?id=com.upft.vr.headtracker.demo

But this app is supported by our module, so no problem. So I think the problem is only with Durovis Dive, which somehow bypass the sensor manager.

2015-09-24 2:30 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

A last update about this issue: so with your code and some more work, I could make the hook work on Eye.getEyeView(), and this fix the issue for most Cardboard apps.

However, Cmoar Cinema Pro v4.3 still isn't supported. I tried to hook into android.hardware.GeomagneticField to check if maybe the app is not using the gyroscope but another orientation provider, but it doesn't work. In fact, all the hooks we implemented are hooking into Cmoar (debug messages are displayed), which means that Cmoar indeed is using the cardboard SDK, but then the hooks are never called (there's never the "Hookx !" message).

What we know using the decompilation of its classes is that Cmoar seems to be using Unity3d, so maybe this issue is affecting all Unity3d apps.

Or maybe not, maybe it's not directly related to Unity3d but just to a third-party library over Unity3d, such as this one:

http://zeissvrone.tumblr.com/post/105593393832/new-unity3d-sdk-11-version-with-integrated-native https://bitbucket.org/vrone/unity3d

https://bitbucket.org/vrone/unity3d#markdown-header-how-do-i-use-the-head-tracking-library-natively-without-unity

I tried to hook on this one and it didn't work, so Cmoar is not using this library, but maybe another one of a similar kind is used.

/EDIT: confirmed, it seems that all Unity3D projects suffer from this issue: https://play.google.com/store/apps/details?id=com.nexus.VRscene

I'm stopping here for today... I'll take a look tomorrow, but if I don't find a way I'll pass for the moment, until there's more documentation :/

2015-09-23 15:28 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

The hooks are working correctly, so this is not an issue with hooking, but only about where to hook and what to do with that hook.

Ok for IRC, I'm connecting right now :)

2015-09-23 12:49 GMT+02:00 Kjos notifications@github.com:

Hooking is a very delicate process. You can create a successful hook for one application and then it might not work in the other. Hooking in an external library is harder than hooking in the system, so it is indeed a challenging task. I do think it's possible having said that, but it is the question whether it is worth the effort.

Do you have time to get on IRC? I'm there now, same place. If you want to continue working on this, I'm willing to help.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-142564172 .

[Kjos]

Take a good look at the Android source: https://github.com/android/platform_frameworks_base/tree/master/core/java/android/hardware . As you can see I originally placed a hook in SystemSensorManager which extends SensorManager. In the Durovis Dive SDK only SensorManager is used. You could hook into SensorManager, but I propose hooking in interfaces SensorListener or similar if possible, as this swats two flies at once.

It might even fix the NDK issue, but I can't tell for sure.

[lrq3000]

Ok that's what I thought, but I needed your confirmation (that using SensorManager bypass SystemSensorManager).

About SensorListener, what method do you advise me to take a look at?

2015-09-24 10:29 GMT+02:00 Kjos notifications@github.com:

Take a good look at the Android source: https://github.com/android/platform_frameworks_base/tree/master/core/java/android/hardware . As you can see I originally placed a hook in SystemSensorManager which extends SensorManager. In the Durovis Dive SDK only SensorManager is used. You could hook into SensorManager, but I propose hooking in interfaces SensorListener or similar if possible, as this swats two flies at once.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-142856749 .

[Kjos]

onSensorChanged in android.hardware.SensorListener and/or android.hardware.SensorEventListener. First parameter is the type of sensor which you compare to GYROSCOPE as usual, second parameter is array of gyro values, also same as before.

These are interfaces, so I'm not sure it will work, but I think it would.

[lrq3000]

Ok thank's, I'm going to try that today :)

2015-09-24 13:01 GMT+02:00 Kjos notifications@github.com:

onSensorChanged in android.hardware.SensorListener and/or android.hardware.SensorEventListener. First parameter is the type of sensor which you compare to GYROSCOPE as usual, second parameter is array of gyro values, also same as before.

These are interfaces, so I'm not sure it will work, but I think it would.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-142896135 .

[lrq3000]

I tried the hooks you suggered, but to no avail. However, when I remove the if condition of the SystemSensorManager hook (ie, it gets applied to all sensors), it seems to tamper with the orientation of the apps that are not supported, but not completely since they can still somehow detect the orientation (although they turn too much with my latest hook).

You can see all the hooks I tried to apply here: https://github.com/lrq3000/XposedJitteryGyroFix/blob/cardboard/GyroscopeNoiseFilter/src/net/kajos/gyronoisefilter/GyroscopeNoiseFilter.java

None of the hooks are getting called in those unsupported apps except SystemSensorManager.

I think I'm done working in this, I don't think we can do much more, they probably use a native/NDK lib which we can't hook onto. However, if you have any idea of a hook to try, I can try quickly.

2015-09-24 13:01 GMT+02:00 Stephen LARROQUE lrq3000@gmail.com:

Ok thank's, I'm going to try that today :)

2015-09-24 13:01 GMT+02:00 Kjos notifications@github.com:

onSensorChanged in android.hardware.SensorListener and/or android.hardware.SensorEventListener. First parameter is the type of sensor which you compare to GYROSCOPE as usual, second parameter is array of gyro values, also same as before.

These are interfaces, so I'm not sure it will work, but I think it would.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-142896135 .

[Kjos]

The hook to getRotation in SensorManager should work with Durovis Dive sdk. I think you shouldn't put in more time, it's kind of as good as it gets.

[lrq3000]

No it does nothing with the Durovis Dive SDK, the getRotation hook never gets called...

Yes I'm already happy with the state of the project. If someone ever finds a way to hook support Durovis Dive SDK, the code is modularized so it's easy to plug in a new hook.

2015-09-25 20:26 GMT+02:00 Kjos notifications@github.com:

The hook to getRotation in SensorManager should work with Durovis Dive sdk. I think you shouldn't put in more time, it's kind of as good as it gets.

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-143314886 .

[Kjos]

Ah, okay, well you already contributed a lot, so that's cool :)

[lrq3000]

Well, I'll watch the project, and if you or someone gets ideas of features or a hook to try to fix this issue, I can quickly code and test but I'm running out of ideas myself :) I'm quite satisfied by the anti-jitter strategies, they work quite well, and the hooks work for most apps so that's great :)

2015-09-25 23:05 GMT+02:00 Kjos notifications@github.com:

Ah, okay, well you already contributed a lot, so that's cool :)

— Reply to this email directly or view it on GitHub https://github.com/Kjos/XposedJitteryGyroFix/issues/4#issuecomment-143354582 .

[lrq3000]

A possible way to anti-jitter: simply cancel the VR tracking movement by applying an inverse calculation:

googlesamples/cardboard-java/issues/27

[Kjos]

That solution might work indeed. Only downside is that you'd have to supply tracking fusion from start to get it working, or hook into the creation of tracking fusion as well, which I figure is more troublesome. I'm on CyanogenMod now, I'm thinking to build a jitter patch for that. Would you be interested in joining on that? It would be more stable than xposed as most are on lollipop which doesn't play nice with xposed. Would also give the ability to tackle the problem at true source giving stable support for the future and all apps using the gyro.