Create layer compsitor for cardboard

[Consti10]

The ability to render layers directly ( omitting a immediate render texture ) is essential for VR on Android devices with limited bandwith and power. For example, oculus built a compositor for most common shaped layers and supports usage on the oculus go. Also, as far as I can tell the gvr library also had support for 2D rectangular layers, but is missing more complex shapes like 360° video.

I would like to propose adding a compositor to the cardboard lib that handles as many layer shapes as possible.

While VDDC is not the most performant solution to render 2D UI layers usw, it would be really easy to adopt (since already open source for cardboard ) and automatically supports more complex shapes like 360° video.

In a next step you could write special shaders for common surfaec shapes like 2D quad or equirectangular. For 2D quads it should be fairly easy ( see ) but I haven't had time to figure out the math yet.

What do you think ? I'd love to contribute. An example renderer for 360° video can be found here: code

[jballoffet]

Thanks for the feature request. We will add this to our backlog. However, if you would like to add these changes, please read CONTRIBUTING.md and follow the process. Patches are welcome. The right place for adding this feature would be within the CardboardDistortionRenderer module, where a variant for _renderEyeToDisplay would be needed, that takes in a list of layer descriptions, each of them containing type, geometry and a texture. We are not familiar with VDDC and are uncertain if it makes sense to add it as a dependency.

[Consti10]

Hello jballoffet, Let me first explain what dependencies / modifications would be needed for adding V.D.D.C to Cardboard:

1. The cardboard undistortion already distorts Vertices instead of Texture coordinates. However, its lens distortion model ( respectively the ViewportParams ) are calculated for a Full Screen OpenGL Viewport in contrast to V.D.D.C, which renders left and right eye in 2 different draw calls with a Viewport of Half the screen. Therefore, I had to make some changes to lens_distortion Most notably adding a new struct for calculating the undistortion ViewportParams in HSNDC ( Half screen normalized device coordinates )

2) V.D.D.C needs a PolynomialRadialInverse to calculate the inverse radial distortion more eficiently. I Have placed all this code in its own folder PolynomialRadialDistortion It adds a PolynomialRadialInverse together with the Math for caclulating it. Note: The old cardboard java library had the appropriate methods for calculating a PolynomialRadialInverse - I mostly only translated google java code to cpp there )

3) For rendering textured geometry with V.D.D.C I have a simple OpenGL shader that can both sample from a 'normal' and 'external' texture and optionally applies V.D.D.C to the vertices: GLProgramTexture You can find more details /helper for V.D.D.C in OpenGL shaders in this file: VDDC.h

Additionally, these Files come with some Header-only dependencies, most notably 'glm' for Matrices and some Helper GLHelper GLBuffer Also they are compiled using c++17 ( I think Cardboard still uses c++11 )

Obviosly, Since some of these files have little in common with the original cardboard files ( For example, I could not get the protobuff library to compile-> therefore I decided to create a new struct MVrHeadsetParams that obtains all device / headset parameters from the gvr library )

However, I can help merging the code into cardboard if you decide to add V.D.D.C to either its core sdk or maybe as an extension.

[Consti10]

Benefits of adding V.D.D.C:

As already mentioned above, using V.D.D.C we can add all layer types that are for example already supported on oculus: unity-ovroverlay Later, depending on their adoption ( for example I expect a high demand for the Equirect layer aka 360* video layer ) we can add customized shaders for these specific layer types.

I already experimented with creating a VrCompositorRenderer.h That holds some VRLayer's and draws all of them using V.D.D.C . I also created an example that contains both a 360° video layer and a 2D ui layer : Renderer360Video

[jballoffet]

Sorry for the delay here. Your proposed changes seem reasonable and valuable for the Cardboard SDK and we would like to take this to the PR phase. Taking into the account the magnitude of the required changes, another good option would be to propose a design first before moving forward with the required changes. Please read CONTRIBUTING.md and follow the process. Thanks for your proposal!

[lifei]

Any update? @Consti10 @jballoffet