Revive Oculus Rift / general VR support [$25]

[Cervator]

I realized there wasn't an issue posted for this yet, just a couple mentions in the forum. Also not minding the idea of putting a small bounty on here that maybe could be added to over time to get this done. After all Oculus is about to go commercial so it is good timing :-)

We supported the Oculus DK1 some time ago at a way older SDK level, as visible here: https://www.youtube.com/watch?v=nNsN_konRbs

The SDK bindings haven't kept up with later releases and then DK2 happened. Occasionally a user would tweak at the support like in http://forum.terasology.org/threads/oculus-issues.890 but even if any code got merged it would go out of date soon enough.

New thread was also just posted: http://forum.terasology.org/threads/support-for-modern-vr-hmds-and-or-motion-controllers.1428

Marking as Contributor friendly with the assumption that anybody who actually has an Oculus Dev Kit would also know how to use it + GSOC candidate for the expanded version of supporting generic VR.

Goal for this issue (to claim the bounty):

• [ ] Update TeraOVR (if needed, alternative available?) - https://github.com/MovingBlocks/TeraOVR
• [ ] Update code to work with latest Oculus Rift SDK and the DK2
• [ ] Associated Javadoc and some basic intro work (forum, readme, or wiki)
• [ ] Any tweaking needed to config options in-game to enable and adjust VR
• [ ] New showcase video to see it all in action

Bonus (required if a GSOC item):

• [ ] Tutorial / better intro doc for module authors who may want to do something extra in VR land (like visual effects or content only applicable to VR somehow)
• [ ] Clean up related rendering systems
• [ ] Add generic support for VR such as via OpenVR
• [ ] Support something beyond the DK2 - CV1 or a different headset

[emanuele3d]

For posterity: I suspect this particular issue is dependent on a considerable restructuring of the rendering engine.

The current Oculus-related code is (too) deeply embedded in the rendering engine. Extracting it and abstracting it, to allow for other HMDs for example, won't be straightforward. This is due to the current rendering engine. No matter its excellent visuals it was not written to be particularly easy to modify or expand.

It might be necessary to implement a radically new architecture, such as the one discussed in #1741 before this issue can be tackled. I might be wrong though and anybody is welcome to try. =)

[Cervator]

Thanks for the thoughts @emanuele3d :-)

Would it almost be worth blessing the DAG pipeline issue for GSOC instead / as a paired item? Although I don't imagine it'll be easy, and mentor availability for 3d wizardry is a bit tricky :-)

[emanuele3d]

I agree with the assessment on mentor availability for the 3d wizardry department... =(

Promoting the DAG-based pipeline idea for the GSOC is certainly worth considering. I would certainly try and implement that before tackling this issue.

But that's just me: it might be theoretically possible for somebody particularly fluent in java/opengl to tackle this issue without major restructuring. I personally feel it would exacerbate the problem.

It might also be possible to restructure the rendering engine in a completely different way from what I proposed in #1741. It would be interesting to hear about radically or even slightly different approaches.

It would certainly be a fairly major project though: perhaps just perfect for the GSOC.

[indianajohn]

Hello,

I have experience porting a closed-source VR application (currently under NDA) to the HTC Vive using OpenVR and an earlier version of the Oculus SDK. I have ordered a personal HTC Vive for my home (I was using a dev kit through my company for the earlier work), and I have some thoughts on the matter and am willing to help with development and testing. Unfortunately, most of my work is in C++/OpenGL, and although I have developed some Android applications using the NDK, I would consider myself a Java novice.

The two main VR toolkits out there have more similarities than differences. The basic idea is:

• Take two 4x4 homogenous transformation matrices from the SDK that define the poses of the left and right eyes in tracking space.
• Take a projection matrix for each eye from the SDK; this is an important step because it ensures that a pixel observed from your left eye's perspective is projected to where your right eye expects.
• For each eye:
  • bind an offscreen framebufffer, and
  • pass projection matrix and pose matrix to your shaders, do whatever you need to inside of there.
• Use a pre-defined distortion pattern defined by the SDK. There is a clear example of this in OpenVR.

Furthermore, OpenVR develops an Oculus back-end. This means that games developed for OpenVR can be played on an Oculus (ideally). In practice, this has been frequently broken in the past.

Unfortunately, OpenVR does not have a ready way of dealing with Java. It is (ironically, given the name) a collection of headers and pre-compiled libraries for Windows, MacOS, and Linux. Naively I would suspect that it would be possible to do one of the following:

• Have a simple C++ program that does nothing more than set up the required OpenGL context, query OpenVR for controller events, and calls renderScene() or somesuch on Terasology.
• Have a simple C++ wrapper for the OpenVR SDK that performs necessary initialization steps and allows the appropriate frame buffers to be bound at the proper times.

The primary difference being where the bulk of the rendering code is written.

Anyway, my equipment doesn't arrive until mid-late next week, and there are a lot of question marks as to how to proceed, but i figured I would kick off the discussion and see if there is any interest. Thanks for your efforts; I see a ton of potential to this game.

[emanuele3d]

@indianajohn, thank you for your interest: work on VR support would certainly be welcome.

What I mentioned in a previous comment still stand however: VR work is dependent on a considerable restructuring of the rendering engine. The good news is: the restructuring is happening. @tdgunes's proposal was accepted within the context of the Google Summer of Code initiative and he's currently working hard to implement a new architecture for the rendering engine. The bad news is: it will takes weeks and possibly the whole allocated period (another two months from now) to get to the point where the code is stable enough for other people to be able to build on it. We certainly want people to build on top of the new architecture, opening the rendering engine is its fundamental goal, but if you start too soon you might find yourself on very shifty grounds.

Given your interest and your level of (Java) experience my suggestion would be first of all to setup your development environment for Terasology. Then I'd recommend you start following tdgunes work. He currently has an open PR (#2367) and the previous one was just merged-in. Then I'd suggest you chip-in on some side-tasks. Think of tdgunes as the guy with the ball running full tilt for a touchdown and you being the crucial teammate getting those pesky opponents out of his way. It would give you the chance to be exposed to important discussions on the current proceedings, learn about the codebase in a focused way and help along the way.

Then, when the time comes and the rendering engine is stable enough, you'd be in a much better position to work on VR.

All that been said, a wrapper for the OpenVR SDK would most likely be needed no matter the new architecture. It just won't be wise to attempt to integrate it in Terasology too soon.

[Cervator]

Heya @indianajohn and welcome! :-)

Yeah we'd appreciate the help, although it may take a little while before we can properly (re)implement VR. Did you check out TeraOVR? Linked in the top post. That's how we originally wrapped the Oculus SDK back in the age of the DK1. I don't really know anything about it and it targets a way outdated SDK level for Oculus (not to mention our rendering has changed and is changing more), but it might be interesting to look at.

I'll echo @emanuele3d on his recommendations, especially on trying out a few smaller issues to get used to the codebase. Then maybe at a good time we can have a little brainstorming/design session on future VR implementation. I'd love to be able to use OpenVR, have been following some and am considering getting their dev kit (got a DK2 at present)

On a vaguely related note we did a proof of concept implementation for the Leap Motion controller once as well, another geeky peripheral. Have you checked out any other hardware projects like that? Any thoughts on how future motion controllers from Oculus, Valve, or others? Way back when I worked on our Leap implementation I wasn't actually sure what to really do with it, so the setup was pretty gimmicky. Main neato part I can imagine working would be something like spell casting with your hands :-)

[indianajohn]

@emanuele3d Thanks for the advice. I'll start reading the bug tracker and commits and see if I can find somewhere to stick my nose in.

@Cervator The Vive is much superior to the Oculus Rift DK2, I can say from experience. The buzzword HTC uses is room-scale VR, and it is an experience head-and-shoulders above the seated experience that Oculus aims for.

Re: TeraVR, I just took a look, and I think something like that would be doable with OpenVR when the time comes closer. Probably using TeraVR as a starting point.

Something like Leap Motion has potential for VR control but it has to be pinned to the Vive reference frame, I think. I've never actually seen a really impressive implementation of a control scheme with it. The controllers on the HTC Vive are completely motion tracked, and I think that is the most promising future value-add of VR. The funnest adaptations are things like shooting games that require painstaking aiming by closing one eye and actually lining up the vector of your shot, or sports games like baseball (batting). So for Teratology, one possibility would be mining blocks by actually striking them with a mining pick that you hold by grasping the controller. At some point in the future combat could also be controlled using the motion controllers.

Anyway, to kick off some ideas on how to make a meaningful standing VR experience (I know these are quite a ways off; I just like talking about it):

• The biggest thing to work around is that translation and rotation of the 3D frame has to be done with some physical analog. If you have a joystick and the frame translates without any stationary reference, it's a one-way ticket to VR sickness. I've handled this by making the user "pull" inertial space like an astronaut grabbing on to a bar in zero-G, but it's not suitable for a game like Terasology. One thing that's occurred to me is creating some sort of platform that looks... vehicular, and gives the user a stationary frame of reference while the camera is translating. Another way that some people do it is a "blink" movement system where you can teleport small distances by selecting your destination by pointing the motion controller at it.
• I'm also not quite sure how to come up with a compelling purpose for the motion controllers that leverages their tracked pose to actually interact with the game world. I mentioned mining blocks through detecting collisions with the controller/pick. In addition to that could be some sort of gesture-based spell casting or skill-based combat.
• It's entirely possible that the VR experience could be an entirely separate facade from those currently under development due to the above reasons.
• Some sort of 3D UI that allows inventory management could also make things a lot cooler/smoother.
• Finally, there are some games on Steam that make use of voice recognition to avoid the need for complicated menu interaction using a VR controller. Something to consider.

[Cervator]

Vive - yeah the DK2 wouldn't stand a chance, nor would it against CV1, I'm looking forward to experience the real thing one day. I played the most with Elite Dangerous and those side menus at that resolution ... aieee! ;-)

Leap - I've tried the combination of it mounted to the front of the DK2. AWESOME effect. I was popping virtual bubbles in front of my face like an idiot, no trouble. Threw around some blocks and stuff too. Still, the tech is there, but actually putting it to use on some sort of gameplay? Trickier.

The matching hand-held controllers getting paired with the headsets are likely going to be better for some stuff since they have a more (well, at all) physical feel, and may even have some feedback like vibration or what not. I am supposed to get a basic VR glove from some crowdfunding campaign, but that project is somewhat in trouble (surprise). Was very curious about the combination of VR + Leap + Glove with individual finger vibration. Imagine drawing a bow, yet be able to drop it then cast a spell :D

Still, it is a brave new world to come up with the gameplay and UIs for all this. Leap Motion has nice Sword Art Online style UI for its setup paired with a VR headset (attaches to your arm then rolls out menus) but I haven't tried it yet.

Looking forward to the potential, one step at a time until then :-)

[Cervator]

Since GitHub supports assigning multiple users now I'm going to start marking who may be interested in an issue. Doesn't mean any one of them will necessarily do it, but I figure it is a good way to highlight who may be relevant :-)

[indianajohn]

I'd like to point out this:

https://github.com/jrbudda/minecrift/tree/vive_1_7_10/src/jopenvr

which appears to be a great base on which to develop an OpenVR plugin for Teratology.

Edit: I guess this is more cleanly applicable: https://github.com/StellaArtois/JOpenVR

Anyway, the point is that there is no need to re-invent the wheel and roll out our own OpenVR Java wrapper.

[Cervator]

Awesome! I had hoped there would be something like that out there, thanks for the find :-)

[AWebb3701]

Vive owner here. Let me know if you need testing done.

[Cervator]

@AWebb3701 - thanks! It'll probably be at least a couple months still before we can hit this, but we can ping you then if available :-)

[indianajohn]

So I had some free time this weekend and reviewed the state of OpenVR support in Java. My conclusion is that it sucks.

• https://github.com/StellaArtois/JOpenVR had some issues compiling and was very incomplete.
• https://github.com/jrbudda/minecrift/tree/vive_1_7_10/src/jopenvr is deeply intertwined with Vivecraft and not able to be used standalone.
• https://github.com/phr00t/jMonkeyVR requires that one adopt an entire game engine as a dependency. (note that Vivecraft is based on this - he pulled jopenvr out of here and used it.)

Anyway, all three of these are unacceptable to use, and it's shameful that nobody (least of all Valve) had addressed the issue, so I took a run at it.

I wrote and tested this over the weekend: https://github.com/indianajohn/jopenvrwrapper

It wraps OpenVR into the following interface:

public class HelloWorld {
        public void run() {
            init();
            loop();
    }

    private void init() {
        // OpenGL and window setup
        // ...
        // OPENVR: Disable v-sync. This is important for VR to get high frame rates.
        glfwSwapInterval(0);
        // ...
    }

    private void loop() {
        // Create the OpenGL context
        GLContext.createFromCurrent();
        // ...

        // OPENVR: create the rendering context for the eyes.
        // This object must be constructed after a valid GLContext exists.
        vrRenderer = new OpenVRStereoRenderer(vrProvider,1280,720);

        // ...
        while ( glfwWindowShouldClose(window) == GL_FALSE ) {
            for (int nEye = 0; nEye < 2; nEye++)
            {
                // OPENVR: bind the FBO associated with the target eye
                EXTFramebufferObject.glBindFramebufferEXT(GL_FRAMEBUFFER_EXT,vrRenderer.getTextureHandleForEyeFramebuffer(nEye));

                 // OPENVR: get rendering transformations
                // Get projection and pose matrices from OpenVR.
                Matrix4f matMVP = vrProvider.vrState.getEyeProjectionMatrix(nEye).mul(vrProvider.vrState.getEyePose(nEye));
                shader.setUniformMatrix("MVP", false, matMVP);

                // ...
                // rendering code

                glEnd();
                // OPENVR: submit frame
                vrProvider.submitFrame();
            }
        }
    }

    public static void main(String[] args) {
        // OPENVR: object initialization.
        OpenVRProvider provider = new OpenVRProvider();
        try {
            // OpenVR: add a controller listener. Could also be this class if we wanted.
            provider.vrState.addControllerListener(new SampleControllerListener());
            Thread vrPoller = new Thread(provider, "vrPoller");
            vrPoller.start();
            SharedLibraryLoader.load();
            HelloWorld app = new HelloWorld();
            app.setVRProvider(provider);
            app.run();
            vrPoller.join();
        } catch (Exception e) {
            System.out.println("Unhandled exception: " + e.toString());
        }
        System.out.println("Exited normally.");"
}

Compling / running that on my machine puts a half-meter psychedelic triangle in the middle of my living room at 90 fps.

I struggled with exactly how much to do for the client, and decided that the interface doesn't need much more trimming down. It only provides controller callbacks, transformation matrices, and VR frame buffer access; the user retains control of their rendering loop.

At the moment, it can be built/run using Grade and integrated by copying some lines from Grade and the relevant classes. I know enough to know that I should do something else, but I figured I should post here as a Java novice and ask advice on how to package this. Use Maven to distribute a jar through a repo?

[Cervator]

Very nice @indianajohn! Thanks for the great update :-)

For our libraries we have build jobs in Jenkins that are configured to publish the resulting built artifact to our Artifactory instance. Then in turn that gives you the ability to declare a dependency in Gradle with group and module name, assuming you include our Artifactory's URL in a repository block.

From there you can then also publish further into Maven Central, jcenter, and so on, so the artifact will be available elsewhere even if our Artifactory is unavailable for some reason.

If you want to play around with that we could set up a test job to build your repo and publish it. Or if you want to go all in we could put it into a MovingBlocks repo and maintain it permanently that way :-)

[Cervator]

Related general news of interest! I just talked to an OSVR rep for a bit about getting Terasology up to speed with an implementation. They're up for some freebie hardware and everything :-)

@indianajohn - do you have a forum account yet or an email you mind sharing with me? Can email to cervator@gmail.com if you're up for it. I'd like to get a little group of VR-interested contributors talking options :-) Maybe I can send you a Slack invite? Would be ideal.

[snowyu]

FYI: The open source ViveCraft is a mod that turns Minecraft into a full-featured VR room-scale experience.

[Cervator]

So it has been about a year since @indianajohn started his successful journey on rejuvenating this topic so it seems like a good time to actually finally close this and call it included with a release ... :D

It might not quite be perfect or heavily used yet, and we broke at least one thing again with #2933, but the way forward is clearer now. #2761 and #2611 cover some related topics. There is more to do but we can capture that in new topics when appropriate :-)