Virtual Reality / 360 Support

rafa8626 commented 7 years ago

From @SeventhCycle on October 7, 2016 0:48

Hello,

There have been 4-5 people in the past week who have messaged me lately asking about VR support in mediaelements.js.

Two of the web players that have been pointed to as examples are: https://krpano.com/ https://developer.jwplayer.com/jw-player/demos/innovation/360-video/

I'm opening this ticket to see:

If there's any plans for this in the near or mid future.
If there isn't, if this is a project that can be potentially sponsored and added to the software.

I haven't seen any open source player tackle VR / 360 degree support, however, I'm thinking it will help VR adoption if there's availability from sources that aren't 100% proprietary.

Thanks for reading.

Copied from original issue: johndyer/mediaelement#1874

rafa8626 commented 7 years ago

We'll need to research but once 3.0 version is out it will be very easy to plug this new video type. It will all depend on other factors and if we can manipulate the video the same way we are doing it now. I'll keep you posted about this

rafa8626 commented 7 years ago

@SeventhCycle After a lot of research and playing with different plugins, I'm planning to use Google's VR View for this. Although there are some limitations indicated in its website, I'd rather go for a script that will be well supported and it's open source. Let me know your thoughts

rafa8626 commented 7 years ago

From @SeventhCycle on November 22, 2016 2:5

@ron666: This looks interesting from what I can see. I'm really interested to see how it turns out.

Here's some things that I have off the top of my head to factor in. Some of these may already be addressed. It's more along the lines of thinking out loud for now since it's at a starting point.

Device support: I see that something like this will work with Google Cardboard, but I'm wondering if it will work with some of the other VR headsets out there, such as the Vive, the Oculus, and possibly the PS (I have no idea if their headset can access a browser at all). Just trying to think about the best way forward on this front.
I see that the storage format on VR is the equirectangular-panoramic format. From my understanding, KRpano supports a different number of different views. I don't know if these are necessary for different devices or not. Forgive me, as I'm a little new to this, too.
That there's a way to determine whether or not a person has a VR headset on. I imagine there will be scenarios where they ONLY want a specific video to play if the person has a VR headset on, and get a regular video otherwise. This may be a boutique request, and that's fine if it is. Just trying to think ahead on it.

I'll ask some of the other people I know with devkits if they have any feedback, and I'll get back to you on this.

Thanks!

rafa8626 commented 7 years ago

@SeventhCycle No problem. As far as I know, the Oculus may work with this, but I don't know about the other 2. For the equirectangular panoramic format I've seen it as the most standard one, and specially if we decide to use MediaElementJS it's more manageable. For the third point, I don't think it's really needed. All the videos should work with headset, but if you don't have one I don't see why we should prevent the users to try to video. Of course might be some limitations but I think we can get away with it.

Let me know your findings, and if this could be a good option, given that is the one I think is very promising and not as intrusive and that fits perfectly with the new approach we are following for 3.x-dev

rafa8626 commented 7 years ago

@SeventhCycle Just to give you an update: I have created the vr-support branch, where you can check the work done with Google VRView. It's still not ready since I'm asking questions to the developers about certain things, but overall it can be used. Events like duration, progress or other html5 native ones are not integrated yet since that's what I'm enquiring. If you wanna check this, please check the test/player-full.html file in that branch. Thanks

rafa8626 commented 7 years ago

From @SeventhCycle on December 24, 2016 1:4

Hi @ron666,

I went ahead and got a Gear VR. I see some promise here!

http://seventhcycle.net/mejsdev/test/player-full.html

When I choose the VR option on this page and play the video, you can put the video fullscreen. When you do, there's a series of options (2d, 360 degree Top Bottom, 180 degree Side by Side, etc) This video seems to be 360 degree Top Bottom. If you put it in that mode, you get to look around and see the Gorilla close-by. It's a neat little effect.

Some notes and questions so far:

Right now, I don't see any ability to click and look around on the desktop. Right now, I just see two videos stacked on top of each other, one for each eye. My guess here is that there's something not implemented yet by the viewer.
When I go into fullscreen, is there any way to pre-specify what kind of video this is? This way, somebody won't have to go in and tell it manually that it's a 360 degree top / bottom video.
I notice as well that if I change the view mode out of 2d, the controls look distorted. I imagine this is because any sort of transform that's needed in order for them to show up in VR isn't happening.

Anyhow, if there's something wrong with my install or build, please let me know and I'll recompile. Or, if you have a public place with your build where isn't working better, please let me know.

Also, please let me know what kind of VR headset you have (if any). I'll see about getting a comparable one.

Thanks and Happy Holidays!

rafa8626 commented 7 years ago

Thanks for your observations. I'll check it and let you know my findings. Happy holidays as well for you too!

rafa8626 commented 7 years ago

OK just download the latest version of the vr-support branch and use the /test/player-full.html file to test this; if you use demos it won't work since the code for VR has been integrated yet in the compiled files. Let me know if it worked for you now. I saw an error message but I don't see it now after the minor fix I set.

rafa8626 commented 7 years ago

From @SeventhCycle on December 26, 2016 20:31

Hi,

The video shows up now within a browser window.

When I put the video fullscreen, I still have to set the settings in order to determine what kind of immersion playback I want.

It might be worth waiting until the WebVR spec (https://iswebvrready.org/) is ready, and Google VR View integrates with it (if possible). I noticed on KRpano 1.19 that I don't have to configure settings for my device when using their new build that uses WebVR. It's able to get the settings for the device and video automatically and serve it to the user without having to do any configuration on the client side.

I don't know if that makes sense or not. Please let me know.

rafa8626 commented 7 years ago

I'll need to check the spec since it's not my area of expertise. Google VR has some additional configuration but I haven't explored it completely. If you find anything that is useful to complete the integration of this piece into MediaElement I'd appreciate it. I'm trying to contact the authors to check some missing features that should be before releasing this, so I'm also waiting for some answers. Thanks for keep looking at this.

poVoq commented 6 years ago

Could this be extended to support the new VR180 format? https://blog.google/products/vr180/how-publish-vr180/

The main difference is that the motion data from the camera is stored in the video itself, which is then used to smooth out the video in VR view to a great extend. Comparing the same videofile playback in a non-VR180 capable player and Googles VR180 app this really makes a huge difference.

mediaelement / mediaelement-plugins

Virtual Reality / 360 Support #4