adapt 3DMol for virtual reality

[dllahr]

@dkoes

Hello Dave

I hope things are well with you. I've reconnected with a former colleague (Steve Brudz, @sbrudz) who is now working on javascript code for virtual reality (e.g. https://github.com/sbrudz/vr-us-population), and I proposed that we work on some biology visualization, trying to combine 3DMol with the VR library / code he's using. I've included @sbrudz 's ideas for how to do this below - short version, implementing a new version of GLViewer. A couple questions -

• how does this approach sound to you?
• should we fork 3DMol to include it, or do it as a separate repo?
  • Or is there another way you could see this being appropriately associated with / using 3DMol?

Thanks in advance, looking forward to hearing your thoughts!

from @sbrudz:

So I've looked over the 3DMol code in more detail and here's what I'm thinking. The VR library I've been using, A-Frame (https://aframe.io/), uses Three.js (https://threejs.org/) to manage all of its WebGL state & rendering. 3DMol implemented all of their own WebGL state management & rendering. I don't think it will play well with A-Frame/Three.js. But the Three.js and 3DMol WebGL wrappers seem to have a similar shape. If we create our own implementation of the GLViewer (http://3dmol.csb.pitt.edu/doc/$3Dmol.GLViewer.html) that uses Three.js under the hood, we should be able to pass that viewer to other 3DMol functions (like $3Dmol.download) to render the molecules. If that works, we can then write an A-Frame component that wraps our version of the GLViewer and see the molecules in VR. How does that approach sound to you? Are there any gotchas that you can think of or things to keep in mind? ... I’m not 100% sure yet that my plan will work, so I’d want to do some experiments first before committing to a particular direction.

[dkoes]

We have a stereoViewer already. It would be great if someone could take the effort to properly parameterize the settings for various VR headsets (e.g. Google Cardboard) and implement a motion-based control scheme.

I do not think it is a good idea to reimplement the rendering engine. It is implemented the way it is for performance reasons.

[sbrudz]

@dkoes Thanks for the fast response. Could you point us to an example of how to use the stereoViewer? I can't find it in the documentation.

[dkoes]

Yes, documentation (and greater configurability) is something that remains on the todo list. Here is an example of it in use: http://3dmol.csb.pitt.edu/tests/auto/generate_test.cgi?test=test96

What we need to do (I think) is adjust the camera settings (in particular the eye offset) to match the VR viewer and, more importantly, the size of the scene. I think scaling so that the bounding sphere of the scene appears about the size of a large beachball held at arms length would be good. There also needs to be programmer control to scale the apparent size.

[sbrudz]

Thanks. Ideally, we'd be able to tie it into one of the existing WebVR frameworks so that we can make use of their headset and controller support. I'll dig in to see if that's a possibility.

[sbrudz]

It seems like all the existing WebVR frameworks (like A-Frame or Babylon.js) either implement their own wrappers around WebGL (babylon) or use another wrapper library (aframe uses three.js). I can't see a way to tie 3DMol.js into one of those frameworks without refactoring it to use a different WebGL wrapper.

So to add WebVR support to 3DMol and keep the current rendering engine, we'd need to tie the stereoViewer into the WebVR API directly. The WebVR API should be able to feed information about the camera settings and matrix transforms to the stereoViewer. I'm still figuring out how the rendering engine works, so I'm not sure yet how difficult that will be.

[dkoes]

Isn't WebVR being deprecated in favor of WebXR which isn't ready for use yet?

[sbrudz]

It is. One of the reasons why I was hoping to be able to tie into A-Frame is that it would act as an abstraction layer to insulate against those upcoming changes to the underlying API. Since that's not an option, maybe it would be better to wait on implementing this until WebXR is more stable.

[andrey-tereshenko]

Cool!