VR

[satoshinm]

Press F4 to enable/disable VR

or Shift-F4 to toggle undistorted stereoscopic rendering

References:

• https://w3c.github.io/webvr/spec/latest/
• https://webvr.info/developers/
• https://www.reddit.com/r/WebVR/
• https://mozvr.com
• https://developer.oculus.com/downloads/legacy/
• https://www.reddit.com/r/oculus/
• https://webvrexperiments.com
• https://github.com/mrdoob/three.js/tree/master/examples/js/effects
• https://www.clicktorelease.com/talks/fullstackfest-2015/slides/#33
• https://github.com/borismus/webvr-boilerplate

[satoshinm]

WebVR seems to still be immature, https://caniuse.com/#feat=webvr is nearly all red. Installed Firefox Nightly and WebVR Enabler Add-on, now known as "WebVR Plus": https://addons.mozilla.org/en-US/firefox/addon/mozilla-webvr-enabler/ - but the https://webvr.info/samples/00-hello-webvr.html sample shows "WebVR supported, but no VRDisplays found." This is using an Oculus DK2 with 0.8.0.0 legacy runtime on macOS 10.12.4 - extending the display natively works, enabled mirroring, can use the built-in scene test demo in OculusConfigUtil (native), not much else. "WebVR Oculus Rift Enabler" extension was also installed in Firefox, disabled extension signing so it could be enabled, but no difference (is it even for WebVR, or for some older spec?). Google Chrome about:flags enabled WebVR, but isn't detected either, although there is a WebVR API Emulation plugin for testing: https://chrome.google.com/webstore/detail/webvr-api-emulation/gbdnpaebafagioggnhkacnaaahpiefil

Since Oculus dropped Mac support, how about testing in Windows? Using Parallels to install Windows 10 in a VM, but can't seem to do HDMI or video monitor passthru, not tested on VMware but from their forums it appears special hardware support would be needed for this to work: https://communities.vmware.com/thread/506477 - so, install Windows 10 natively using Boot Camp. This allows installing the latest Oculus Rift setup package: https://www3.oculus.com/en-us/setup/ - instead of legacy, but can't set it up because although everything else meets spec (memory, CPU, OS), the GPU does not (AMD Radeon R9 M295X in this iMac. Not unexpected, but would've hoped the Windows support software would continue support for DK2, as its minimum hardware requirements, but not so. Back to the 0.8.0.0 "legacy" Oculus, installed on Windows 10, scene test demo works, but similar setup for Firefox and Chrome doesn't fix the WebVR samples either. Nothing appears to be gained using Windows over macOS with the Oculus Rift DK2, at least functionality-wise, so I'm back to Mac.

Using mirroring (Displays preferences, rotate 90º, mirror) / extended displays - as opposed to "direct mode" (wasn't ever supported on Mac and Linux anyways), with the headset connected over an HDMI/Thunderbolt adapter + USB, and the VR content rendering side-by-side stereo scenes fullscreen, seems to be the most practical means of currently delivering VR content at this time. (*)

(*) Meaning, not using/requiring WebVR or any VR API but pure WebGL or native GL, "manual VR"

From https://en.wikipedia.org/wiki/Google_Cardboard#Viewer_assembly_and_operation:

Google Cardboard–compatible app splits the smartphone display image into two, one for each eye, while also applying barrel distortion to each image to counter pincushion distortion from the lenses.

Paper on the mechanics of this barrel distortion: http://www.vassg.hu/pdf/vass_gg_2003_lo.pdf "Applying and removing lens distortion in post production"

[satoshinm]

First things first, the side-by-side rendering needs a correct viewport and aspect ratio, not currently the case (left is squashed):

Rendering half the viewport, same on each side, is passable (no 3d, but can view through VR HMD) but has several problems: crosshairs split, targeting is all off to the side (movement, break/place):

[satoshinm]

[satoshinm]

Looking to three.js as an example. Found this, promising: http://stackoverflow.com/questions/16798412/how-to-use-oculusrifteffect-js-on-the-webgl-interactive-cubes-examples#27346665 - but the example it links to is 404: https://github.com/mrdoob/three.js/blob/master/examples/webgl_geometry_minecraft_oculusrift.html and so are all the other links from that user in 2013, link rot. https://github.com/carstenschwede/RiftThree exists but I'd like to see the "webgl_geometry_minecraft_oculusrift.html" demo. OculusRiftEffect can be found here: https://github.com/vladikoff/voxel-oculus-vr/blob/master/OculusRiftEffect.js. This seems to be the original demo: http://www.html5canvastutorials.com/libraries/Three/Three/examples/webgl_geometry_minecraft_oculusrift.html. It doesn't have pointer lock or fullscreen? but it does show the barrel distortion:

Here are all the latest three.js demos: https://threejs.org/examples/

This page shows how to get OculusRiftEffect working with the DK2, instead of DK1: http://laht.info/tag/oculusrifteffect-js/

    // Specific HMD parameters
    var HMD = (options && options.HMD) ? options.HMD: {
        // Parameters from the Oculus Rift DK2
        hResolution: 1920, // <--
        vResolution: 1080, // <--
        hScreenSize: 0.12576, // <--
        vScreenSize: 0.07074, // <--
        interpupillaryDistance: 0.0635, // <--
        lensSeparationDistance: 0.0635, // <--
        eyeToScreenDistance: 0.041,
        distortionK : [1.0, 0.22, 0.24, 0.0],
        chromaAbParameter: [ 0.996, -0.004, 1.014, 0.0]
    };

included in three.js r69, but again this is 404: https://github.com/mrdoob/three.js/blob/master/examples/js/effects/OculusRiftEffect.js. Updated upstream here for DK2: https://github.com/mrdoob/three.js/commit/fc73dbca5a1595e24277f011f29c1f9be7fb984a. Originally added to three.js in https://github.com/mrdoob/three.js/pull/3025. This page is also now 404: https://threejs.org/examples/webgl_effects_oculusrift.html (being diagnosed in https://github.com/mrdoob/three.js/issues/4902). Cloning to dig where it was removed.

---

OculusRiftEffect and OculusControls were removed from three.js on January 20th, 2015: https://github.com/mrdoob/three.js/commit/d22bb22db0fea6a256c2417ee34118cb9c2df6b9 "Examples: Removed OculusControls/OculusRihtEffect. Use WebVR instead." - but I can't use WebVR with the Oculus DK2, it is not supported. Not going to use WebVR for now, instead, apply distortion through the shader, this file shows how: https://github.com/mrdoob/three.js/blob/36565aa86a44d02cdb9c8af4ba91816928180fab/examples/js/effects/OculusRiftEffect.js

[satoshinm]

Added shaders, they compile and link but need to be hooked up. The basic strategy is to render to a texture2D (three.js calls a ShaderMaterial), twice (one for each eye) with different parameters (uniforms, viewport, matrix).

[satoshinm]

Starting from the basics, not even applying the barrel distortion shader: there is a problem, in fullscreen (no VR, but hit F11) the screen is too tall (and there are black bars on the sides) -- native only:

perhaps related to the 90º rotation required setting:

this is how it is supposed to look, from OculusConfigUtil scene demo:

[satoshinm]

Native build, first F11 quickly enters fullscreen with correct dimensions. Press F11 to go back to windowed mode, ok. But then press F11 again to reenter fullscreen, and it enters it differently: fades in slowly, and has the wrong dimensions. Using the green bubble to toggle fullscreen/windowed mode always works. Am I using glfwSetWindowMonitor() wrong?

---

Testing now, I only always get the second behavior(?). init_fullscreen_monitor_dimensions() gets the modes, chooses the last, these are:

fullscreen_monitor = 0x7fa81b201590
mode 0: 1138 x 640
mode 1: 720 x 1280
mode 2: 948 x 1080
mode 3: 1422 x 800
mode 4: 900 x 1440
mode 5: 1050 x 1680
mode 6: 1820 x 1024
mode 7: 1920 x 1080
mode 8: 1920 x 1080
mode 9: 1920 x 1080
mode 10: 1080 x 1920
mode 11: 1200 x 1920
mode 12: 1440 x 2560
mode 13: 1600 x 2560
fullscreen_monitor = 0x7fa81b201590
width, height = 1600 x 2560
fullscreen_monitor = 0x7fa81b201590
width, height = 1600 x 2560

This is when the DK2 is plugged in. Display settings shows the native resolutions:

---

The last mode isn't 1920x1080. 7-9 are 1080p, only differing in refresh rate:

mode 7: 1920 x 1080, bits 8 8 8, 60 Hz
mode 8: 1920 x 1080, bits 8 8 8, 72 Hz
mode 9: 1920 x 1080, bits 8 8 8, 75 Hz

choose not last mode (1600x2560 = 4,096,000, versus 1920x1080 = 2,073,600?) but highest width and refresh rate? Or what about: http://www.glfw.org/docs/latest/group__monitor.html#gafc1bb972a921ad5b3bd5d63a95fc2d52 glfwGetVideoMode() - returns current mode. Then fullscreen to current mode, don't change it.

[satoshinm]

Fixed that by using the current monitor resolution, instead of second-guessing what to use by selecting the highest/last, which may not have the correct rotation aspect ratio or refresh rate. The player can change their monitor resolution in their OS settings before starting the game if they want something higher. F4/vr mode after F11/fullscreen now shows the correct viewports and resolutions, as evidenced by the pixelated text (half of 1080p to each viewport):

Now to port the rest of render(): https://github.com/mrdoob/three.js/blob/36565aa86a44d02cdb9c8af4ba91816928180fab/examples/js/effects/OculusRiftEffect.js#L168

Render to a texture, is this what OculusRiftEffect formerly in three.js is doing in the hood, as a ShaderMaterial? http://www.opengl-tutorial.org/intermediate-tutorials/tutorial-14-render-to-texture/

[satoshinm]

Began to implement rendering to texture, but not complete, crashing on the glFramebufferTexture(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, g->vr.texture, 0); call, needs further investigation. Further reading:

• http://www.opengl-tutorial.org/intermediate-tutorials/tutorial-14-render-to-texture/
• https://developer3.oculus.com/documentation/pcsdk/latest/concepts/dg-render/
• https://forums.oculus.com/community/discussion/comment/54165/
• https://www.khronos.org/registry/OpenGL-Refpages/es2.0/xhtml/glRenderbufferStorage.xml
• https://www.khronos.org/registry/OpenGL-Refpages/es2.0/xhtml/glFramebufferRenderbuffer.xml
• https://news.ycombinator.com/item?id=14257818 https://blog.ycombinator.com/how-to-get-into-vr/

---

Also, 086e87d82ff12897926461d6feda30a87d4c04ce fullscreening to current video mode instead of highest, broke fullscreening to Retina display without DK2 connected. update: fixed by special-casing

---

Hint from http://stackoverflow.com/questions/23550052/why-is-glframebuffertexture-a-null-pointer-when-glew-arb-framebuffer-object-is-n: glFramebufferTexture() was null, try glFramebufferTexture2D() it is better supported

[satoshinm]

glCheckFramebufferStatus(GL_FRAMEBUFFER) is supposed to return GL_FRAMEBUFFER_COMPLETE if ready, but it is returning 36059 which is GL_FRAMEBUFFER_INCOMPLETE_DRAW_BUFFER, with what I have so far. Something isn't set up correctly.

Example for reference: https://github.com/opengl-tutorials/ogl/tree/master/tutorial14_render_to_texture (build with cmake) - this works, launches fine. Adapting the code to not setup every render loop iteration, get a different error: 36055 = FRAMEBUFFER_INCOMPLETE_MISSING_ATTACHMENT_EXT. And note that opengl-tutorials calls glFramebufferTexture(), not glFramebufferTexture2D().

---

If I change https://github.com/opengl-tutorials/ogl/blob/master/tutorial14_render_to_texture/tutorial14.cpp#L186:

    // Set "renderedTexture" as our colour attachement #0
    //glFramebufferTexture(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, renderedTexture, 0);
    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, renderedTexture, 0, 0);

then tutorial14 fails with the same 36055 (FRAMEBUFFER_INCOMPLETE_MISSING_ATTACHMENT_EXT). Oh, the arguments are wrong, see: https://www.khronos.org/registry/OpenGL-Refpages/es2.0/xhtml/glFramebufferTexture2D.xml

    glFramebufferTexture2D(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_TEXTURE_2D, renderedTexture, 0);

this fixes tutorial14, but NetCraft now fails 30654 GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT different error, hint: http://stackoverflow.com/questions/36263693/glcheckframebufferstatus-always-returns-36054#36306241

There is no implementation of OpenGL / OpenGL ES supports alpha or luminance textures as color-renderable.

[satoshinm]

The working example https://github.com/opengl-tutorials/ogl/blob/master/tutorial14_render_to_texture/tutorial14.cpp is written for a different version of OpenGL, glfwWindowHint is used to request OpenGL 3.3 core. This is where #version 330 core in its shader programs come in. http://www.glfw.org/docs/latest/window.html#window_hints_ctx

3.2 core also works, but 3.1 and 3.0 fail to open the window.

OpenGL: GLFW_CONTEXT_VERSION_MAJOR and GLFW_CONTEXT_VERSION_MINOR are not hard constraints, but creation will fail if the OpenGL version of the created context is less than the one requested. It is therefore perfectly safe to use the default of version 1.0 for legacy code and you will still get backwards-compatible contexts of version 3.0 and above when available.

What is the default? Craft doesn't specify any:

While there is no way to ask the driver for a context of the highest supported version, GLFW will attempt to provide this when you ask for a version 1.0 context, which is the default for these hints.

Emscripten "targets the WebGL-friendly subset of OpenGL ES 2.0", http://kripken.github.io/emscripten-site/docs/porting/multimedia_and_graphics/OpenGL-support.html . Note activation:

Additionally, in WebGL, unlike in desktop or mobile OpenGL, extensions must be activated first before the features they expose take effect.

but I'm testing this on desktop (native) first, before porting to web. There are some more examples at https://github.com/kripken/emscripten/tree/master/tests/glbook.

---

http://stackoverflow.com/questions/3613889/gl-framebuffer-incomplete-attachment-when-trying-to-attach-texture#3614043 discusses the meaning of GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT, an incomplete texture - but I have set the min filter etc. Tracing through the calls, first it is set to GL_FRAMEBUFFER_COMPLETE (0x8CD5 = 36053) right after glGenFramebuffers, then GL_FRAMEBUFFER_INCOMPLETE_MISSING_ATTACHMENT (0x8CD7 = 36055) all up to the glFramebufferTexture2D() call, then it goes to GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT (0x8CD6 = 36054).

---

Copied and pasted the exact same code from tutorial14 example, same failure - turns out, was passing 0 width and height to the texture! This caused GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT. Changed to initialize with glfwGetFramebufferSize() (normally, it is initialized in the render loop), and now no longer fails.

[satoshinm]

We have an offscreen framebuffer, now to use it as a texture to the barrel distortion shader, like this: https://github.com/mrdoob/three.js/blob/36565aa86a44d02cdb9c8af4ba91816928180fab/examples/js/effects/OculusRiftEffect.js#L168 and https://github.com/opengl-tutorials/ogl/blob/master/tutorial14_render_to_texture/tutorial14.cpp#L223

---

Geometry: OculusRiftEffect.js uses a three.js plane geometry, of 2x2 - specifically:

    var mesh = new THREE.Mesh( new THREE.PlaneBufferGeometry( 2, 2 ), RTMaterial );

ogl/tutorial14 uses a quad from -1,-1 to +1,+1, essentially same thing:

    // The fullscreen quad's FBO
    static const GLfloat g_quad_vertex_buffer_data[] = { 
        -1.0f, -1.0f, 0.0f,
         1.0f, -1.0f, 0.0f,
        -1.0f,  1.0f, 0.0f,
        -1.0f,  1.0f, 0.0f,
         1.0f, -1.0f, 0.0f,
         1.0f,  1.0f, 0.0f,
    };

---

Started to implement it, but not complete, black screen, likely related to textures, need to debug.

[satoshinm]

With the glClearColor set to green, and this clause set to return red instead of black:

    if (any(bvec2(clamp(tcBlue, vec2(0.0,0.0), vec2(1.0,1.0))-tcBlue))) {
        gl_FragColor = vec4(1.0, 0.0, 0.0, 1.0);
        return;
    }

we see the left eye (only shown here) is hitting the if condition all the time:

Need to find out what the condition is for and why it is getting hit. tcBlue is calculated from:

    vec2 rBlue = rvector * (chromAbParam.z + chromAbParam.w * rSq);
    vec2 tcBlue = (lensCenter + scale * rBlue);

[satoshinm]

There we go... it was a simple integer division error when porting the JavaScript into C! We now have the barrel distortion, just need to wire up the other eye:

---

Each eye can be rendered individually, but not together, may need to separate out textures?

[satoshinm]

Just needed to not clear, now we have both eyes:

but when viewed through the VR headset, it looks like a flat sphere - because of the missing matrix translation of "interpupillary distance" needed to achieve the 3d effect. Described on https://developer3.oculus.com/documentation/pcsdk/latest/concepts/dg-render/:

When using the Rift, the left eye sees the left half of the screen, and the right eye sees the right half. Although varying from person-to-person, human eye pupils are approximately 65 mm apart. This is known as interpupillary distance (IPD). The in-application cameras should be configured with the same separation.

Note: This is a translation of the camera, not a rotation, and it is this translation (and the parallax effect that goes with it) that causes the stereoscopic effect. This means that your application will need to render the entire scene twice, once with the left virtual camera, and once with the right.

This is how OculusRiftEffect did it: https://github.com/mrdoob/three.js/blob/36565aa86a44d02cdb9c8af4ba91816928180fab/examples/js/effects/OculusRiftEffect.js#L127

        // Compute camera projection matrices
        var proj = (new THREE.Matrix4()).makePerspective( fov, aspect, 0.3, 10000 );
        var h = 4 * (HMD.hScreenSize/4 - HMD.interpupillaryDistance/2) / HMD.hScreenSize;
        left.proj = ((new THREE.Matrix4()).makeTranslation( h, 0.0, 0.0 )).multiply(proj);
        right.proj = ((new THREE.Matrix4()).makeTranslation( -h, 0.0, 0.0 )).multiply(proj);

        // Compute camera transformation matrices
        left.tranform = (new THREE.Matrix4()).makeTranslation( -worldFactor * HMD.interpupillaryDistance/2, 0.0, 0.0 );
        right.tranform = (new THREE.Matrix4()).makeTranslation( worldFactor * HMD.interpupillaryDistance/2, 0.0, 0.0 );

Ported in g->vr.left/right.proj/transform, but needs to be computed within render_scene() in the model view projection matrix.

Craft uses set_matrix_3d() in render_chunks() and elsewhere, implemented in src/matrix.c, to call mat_perspective() if in perspective view or mat_ortho if in orthographic view. Add the ±h translation here?

[satoshinm]

Added a toggle for skipping the barrel distortion effect (Shift-F4), useful for testing because it uses the same pipeline but otherwise omits the effect. I think with WebVR you're supposed to render without distortion and let it do it for you. But for testing, without WebVR support, here is the same scene rendered in three different modes:

Normal:

VR (pressed F4):

Undistorted VR (press Shift-F4):

Clearly have some work to do with viewports, and depth buffers face culling. But at least the distortion works now.

[satoshinm]

drawBuffers() requires the WEBGL_draw_buffers extension of WebGL 2, so to support platforms without it (but without VR) need to conditionalize init_vr(). http://webglstats.com/webgl/extension/WEBGL_draw_buffers says right now 59% of users support WEBGL_draw_buffers (76% on desktop, very few on other platforms).

[satoshinm]

Open problems: why does OculusRiftEffect do two translations for the IPD, one in projection matrix and another in the translation matrix, do I need both? And also the viewport dimensions (try in Shift-F4 mode, should match the viewport in non-VR, not be shifted or offset).

[satoshinm]

Some issues (viewport seems to inconsistently shift offset?) but it basically works, at least good enough to merge: