GVRf AR integration.

[lcbasu]

Hi @thomasflynn ,

We've proposed changes to the existing GVRf library to allow third party AR libraries, such as Vuforia to use GVRf as their rendering api. Some small changes are required to the core of GVRf to integrate AR. Here [https://github.com/SRIB-GRAPHICS/GearVRf-AR] is the repo with all those changes and a simple sample app to test the integration.

Please provide your feedback on this integration and once the initial issues are taken care of, we will create a separate branch and raise a pull request for the integration.

Thanks.

[phi-lira]

Is it possible to create subclasses of material for these new AR features? Like GVRARMaterial and ARMaterial? I think this would be better as we would make AR code transparent to regular SDK use and we would avoid all the if (ar) blocks.

Also, is there a way to avoid setting MVP (and similar) directly by passing to SDK the AR object transform and setting an AR camera? If so, no changes would be required on the core SDK regarding matrices.

[thomasflynn]

We have a gvr-vuforia sample that show how to pass a matrix of a market to a sceneobject. I think it is just setModelMatrix(). Check the sample. I'm on PTO 'til friday, i'll take a closer look at the code then. Meanwhile @chenchao, can you ask Gaurav to take a look at this? Thanks.

[chenchao1407]

OK.

[gaurav-mcl]

Hello @lokeshguddu,

There are jni errors while building Framework. Seems to be missing ovr_mobile_sdk directory.

/home/gaurav001.s/This_Folder_Is_Backedup/Downloads/adt-bundle-linux-x86_64-20140321/ndk/ndk-build -j16 OVR_DEBUG=0 NDK_DEBUG=0 Android NDK: WARNING: Ignoring unknown import directory: ../../ovr_mobile_sdk
jni/Android.mk:36: ../../ovr_mobile_sdk/cflags.mk: No such file or directory Android NDK: jni/Android.mk: Cannot find module with tag 'LibOVR/Projects/Android/jni' in import path
jni/Android.mk:132: *\ Android NDK: Aborting. . Stop. Android NDK: Are you sure your NDK_MODULE_PATH variable is properly defined ?

[gaurav-mcl]

Sorry, please ignore previous comment. I didn't know that ovr_mobile_sdk referred to Oculus mobile sdk.

[gaurav-mcl]

The sample app ARVRChess complains of a missing lib EFXLib on the build path.

[lcbasu]

Hi @gaurav-mcl ,

Thanks for bringing this to our notice.

We found out the issue and will be pushing the changes soon.

[eesha19]

Hi @gaurav-mcl , Here are the changes that we made (https://github.com/SRIB-GRAPHICS/GearVRf-AR/commit/bdfaebcc3377129528a71e1a6948916eb65f2187). If the EFXLib (missing) is still visible in the Java Build Path, please remove it, use the changed .classpath file, and then try building. Hope it runs fine now.

[thomasflynn]

I took a deeper look at this over the weekend. This appears to be a pretty heavy change. Especially with introducing a AR vs VR mode. It appears as though AR mode is needed for two things:

1) passing a vuforia matrix to a SceneObject 2) using the vuforia fbo for rendering the pass-through camera.

Is there something else I'm missing?

We already have an example of #1 in the gvr-vuforia sample. And we have an example of #2 in some other code we have that isn't public. If we provide an example of using vuforia for the pass-through camera, would that provide all you need?

[gaurav-mcl]

In addition to Tom's comment, I am also having difficulty running the app on the device. I am using a Note 4 device. The app crashes with the below logcat. The only changes I made to your sample app are: (1) remove EFXLib from the classpath, (2) changed the path of Vuforia.jar to match the location on my machine. It builds and installs fine but crashes right on startup. Just to add, I am starting the app only after plugging the device into the headset as requested by the app.

Logcat

I/App (30912): 0xb4ef5000 msg: intent I/art (30912): Rejecting re-init on previously-failed class java.lang.Class I/art (30912): Rejecting re-init on previously-failed class java.lang.Class D/AndroidRuntime(30912): Shutting down VM E/AndroidRuntime(30912): FATAL EXCEPTION: main E/AndroidRuntime(30912): Process: com.qualcomm.vuforia.samples.VuforiaSamples, PID: 30912 E/AndroidRuntime(30912): java.lang.NoClassDefFoundError: Failed resolution of: Lcom/qualcomm/vuforia/samples/SampleApplication/SampleApplicationSession; E/AndroidRuntime(30912): at org.gearvrf.ARVuforia.AppActivity.onCreate(AppActivity.java:80) E/AndroidRuntime(30912): at android.app.Activity.performCreate(Activity.java:6289) E/AndroidRuntime(30912): at android.app.Instrumentation.callActivityOnCreate(Instrumentation.java:1119) E/AndroidRuntime(30912): at android.app.ActivityThread.performLaunchActivity(ActivityThread.java:2665) E/AndroidRuntime(30912): at android.app.ActivityThread.handleLaunchActivity(ActivityThread.java:2779) E/AndroidRuntime(30912): at android.app.ActivityThread.access$900(ActivityThread.java:179) E/AndroidRuntime(30912): at android.app.ActivityThread$H.handleMessage(ActivityThread.java:1462) E/AndroidRuntime(30912): at android.os.Handler.dispatchMessage(Handler.java:102) E/AndroidRuntime(30912): at android.os.Looper.loop(Looper.java:145) E/AndroidRuntime(30912): at android.app.ActivityThread.main(ActivityThread.java:5974) E/AndroidRuntime(30912): at java.lang.reflect.Method.invoke(Native Method) E/AndroidRuntime(30912): at java.lang.reflect.Method.invoke(Method.java:372) E/AndroidRuntime(30912): at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1388) E/AndroidRuntime(30912): at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1183) E/AndroidRuntime(30912): Caused by: java.lang.ClassNotFoundException: Didn't find class "com.qualcomm.vuforia.samples.SampleApplication.SampleApplicationSession" on path: DexPathList[[zip file "/data/app/com.qualcomm.vuforia.samples.VuforiaSamples-1/base.apk"],nativeLibraryDirectories=[/data/app/com.qualcomm.vuforia.samples.VuforiaSamples-1/lib/arm, /vendor/lib, /system/lib]] E/AndroidRuntime(30912): at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:56) E/AndroidRuntime(30912): at java.lang.ClassLoader.loadClass(ClassLoader.java:511) E/AndroidRuntime(30912): at java.lang.ClassLoader.loadClass(ClassLoader.java:469) E/AndroidRuntime(30912): ... 14 more E/AndroidRuntime(30912): Suppressed: java.lang.NoClassDefFoundError: com.qualcomm.vuforia.samples.SampleApplication.SampleApplicationSession E/AndroidRuntime(30912): at dalvik.system.DexFile.defineClassNative(Native Method) E/AndroidRuntime(30912): at dalvik.system.DexFile.defineClass(DexFile.java:226) E/AndroidRuntime(30912): at dalvik.system.DexFile.loadClassBinaryName(DexFile.java:219) E/AndroidRuntime(30912): at dalvik.system.DexPathList.findClass(DexPathList.java:321) E/AndroidRuntime(30912): at dalvik.system.BaseDexClassLoader.findClass(BaseDexClassLoader.java:54) E/AndroidRuntime(30912): ... 16 more E/AndroidRuntime(30912): Suppressed: java.lang.ClassNotFoundException: com.qualcomm.vuforia.samples.SampleApplication.SampleApplicationSession E/AndroidRuntime(30912): at java.lang.Class.classForName(Native Method) E/AndroidRuntime(30912): at java.lang.BootClassLoader.findClass(ClassLoader.java:781) E/AndroidRuntime(30912): at java.lang.BootClassLoader.loadClass(ClassLoader.java:841) E/AndroidRuntime(30912): at java.lang.ClassLoader.loadClass(ClassLoader.java:504) E/AndroidRuntime(30912): ... 15 more E/AndroidRuntime(30912): Caused by: java.lang.NoClassDefFoundError: Class not found using the boot class loader; no stack available V/ApplicationPolicy( 978): isApplicationStateBlocked userId 0 pkgname com.qualcomm.vuforia.samples.VuforiaSamples

[gaurav-mcl]

OK, I have your sample app running with some effort. For some reason, the Vuforia.jar was not getting packaged with the apk and hence the app was crashing due to not finding the Vuforia classes.

So the app shows a camera passthrough using GVRF and you can track a marker by displaying teapot on it.

As Tom mentioned, we have a app of our own which integrates Vuforia with GVRF. It does not have camera passthrough yet. But we think we can enable that without making deep changes in the GVRF. I will revert with our modified sample app soon.

[gaurav-mcl]

Hello @lokeshguddu,

I created a pull request https://github.com/Samsung/GearVRf/pull/212 which modified the gvr-vuforia sample to enable camera passthrough. Can you please review this pull request to see if this is the functionality you are looking for ?

The modifications I made may seem overwhelming but most of them are related to cleaning up the way Vuforia initialization is done in the app and also removing some mesh and textures that we think are not relevant to this app. What may be of interest to you is how the vuforia camera pass through is enabled: please look at the methods createCameraPassThrough() and onStep() in VuforiaSampleScript.java.

This app achieves the same functionality as your proposed changes: showing a camera pass through in GearVR using Vuforia and be able to track a marker. But the thing worth mentioning is that there are absolutely no changes made to the GearVR Framework. All changes are in the app only.

When you run the app, a few times, you may only see the teapot and not the camera pass through video background. Just close the app and rerun. And you should be able to see the video as well. This is probably some minor bug in the Vuforia pipeline which we can investigate in future. But for your purposes, this should be fine.

[lcbasu]

@eesha19 can you look into the pull request made by @gaurav-mcl. Thanks.

[gaurav-mcl]

Hello @lokeshguddu, @eesha19 do you have any comments on the pull request https://github.com/Samsung/GearVRf/pull/212 that I created a few days ago ?

We made some minor improvements to the app since then. So please check it out. My colleagues here have already reviewed the pull request. So if this is serving your needs, we can merge it in the main branch.

[lcbasu]

Hi @gaurav-mcl ,

We will get back to you with the review asap.

[eesha19]

Hi @gaurav-mcl ,

We have updated our framework setup to have minimum number of changes now. The only major thing is passing the following:

• isAR flag
• MVP Matrix

• Viewport Offsets.

  If we avoid doing these changes in the framework, there are significant issues as seen in the recently merged app - Viewport settings will be incorrect with the tracked object going outside the preview window. The aspect ratio can't also be matched to provide a seamless direct AR type experience. Another problem with that app is that there is no control on when to display the virtual object - the virtual object is visible even in the splash screen which is incorrect. The frame rates in our implementation are much superior as well.

[gaurav-mcl]

I will review the change.

Just one quick comment .... the statement "Another problem with that app is that there is no control on when to display the virtual object - the virtual object is visible even in the splash screen which is incorrect. " is not accurate.

This was an issue in the initial few commits of the app that we merged recently. But in the final commit, you only see the virtual object when the marker is detected in the camera viewpoint. So there should be no virtual object during the splash screen.