Native support of OVR Overlay

[lightsailvr]

Is your feature request related to a problem? Please describe. Oculus SDK has introduced the ability to play video on an Overlay layer, bypassing the eye buffer and displaying the video much sharper and without as much jitter as before. I would like to see AVPro support rendering to this layer as it greatly improves image quality.

Describe the solution you'd like A demo scene showing how to render both a flat video and 360/180 spherical video to the OVR Overlay

Describe alternatives you've considered I have tried to utilize Unity's built in video play to accomplish this, but it's no match feature for feature with AVPro, especially when it comes to streaming.

Additional context Video encoding settings being used: https://creator.oculus.com/blog/encoding-high-resolution-360-and-180-video-for-oculus-go/

More information about OVR Overlay: https://developer.oculus.com/documentation/unity/latest/concepts/unity-ovroverlay/

[andriworld]

Hi @lightsailvr, I got this working by taking the texture from AVPro (as soon as it's not null) and assigning it to OVR Overlay: ovrOverlay.textures[0] = mediaPlayer.TextureProducer.GetTexture();

It works great! What would be even better is if we could assign the pointer to the native android texture, but AVPro doesn't seem to expose it 'ovrOverlay.externalSurfaceObject = ???'

[AndrewRH]

Interesting! I didn't know you could do that....we'll have to expose the surface pointer so you can try it!

[Ste-RH]

@andriworld Is it one of these you are after?

https://developer.android.com/reference/android/view/Surface

[blanxdev]

@RenderHeadsSte I think in this example you can see how to use the external surface properly, but I am not good at native coding.

https://developer.oculus.com/blog/tech-note-animated-loading-screen/

[Ste-RH]

Yeah, I already looked at that sample. It didn't give me the answer.

Before we can make a call on whether we will expose access, we need to know what object you want.

[blanxdev]

One way is to let the video player write to a surface texture or whatever it is called. In the Oculus video player example they provide a surface to the player.

` exoPlayer = ExoPlayerFactory.newSimpleInstance(new CustomRenderersFactory(context), trackSelector);

            exoPlayer.setVideoSurface( surface );

            // Prepare the player with the source.
            exoPlayer.prepare(videoSource);`

And I think the surface gets created in the OVROverlay.cs component which then is used in a overlay layer. This should be the most efficient approach.

"OVR Overlay – Used as a surface on which ExoPlayer plays the video on Mobile platforms.

Of note is the Android-only Is External Surface check box. When checked, this allows for an Android external surface to be controlled by the Asynchronous TimeWarp (ATW) layer. In this sample, this allows the Android ExoPlayer video player surface to be run as an ATW layer, avoiding redundant texture copies to save memory and cycles."

What @andriworld proposed is a different approach. There he wants to use it as a texture I think and just wants a pointer to the texture data. But not sure if I got it right.

[Ste-RH]

Oki, so 'surface' there is the one I linked to (android.view.Surface).

We will have a chat about this internally and make a decision about exposing it.

[blanxdev]

Looking forward. Hopefully then it is possible to do playback on Go without any framedrops. We are trying to playback 60fps footage, but having frame drops every quarter or half second which disturbs the experience. With Oculus Gallery it plays without these drops.

[Ste-RH]

Roger that. Your use case certainly gives weight exposing it.

[blanxdev]

Let me know if you need somebody to test it. Also I can provide you a video file that produces the frame drops on our side.

[blanxdev]

Just found out that this also exists for GoogleVR https://docs.unity3d.com/Manual/VRDevices-GoogleVRVideoAsyncReprojection.html

[lightsailvr]

About to finally give this a whirl now that this video player project is due. Been struggling with the Native video player used in the oculus SDK because it doesn’t have any calls or hooks to check to see if the video is playing for example. This makes it impossible to sync our external quad binaural audio. Any update on whether this feature is coming?

[Drugges]

I am looking forward to this as well. This would (could) be a great feature!

[robertncoomber]

I was able to successfully render a texture from AVPro to an OVR overlay layer. It is rendering a stereo 4k 360 video using Oculus Integration 1.44 and AVPro 1.10.1. I took some demo code from Oculus's Stereo180Video.unity sample scene to properly set up the Overlay layer.

I feel like the video is much sharper on the OVR overlay layer. However, I am noticing some strange color artifacts. Playback on the OVR Overlay layer is much brighter. In addition there seems to be some compression happening on the colors. I am not a compression expert but there seems to be less colors rendering.

I've only spent a couple hours playing with it so there might be something I am missing or set up incorrectly. I am going to continue working with it to try to get it rendering properly.

But being able to assign it to an external texture in theory would produce even better results. I would certainly use it.

Edit: The color is fixed by switching color space to gamma. It is properly rending in stereo and the video looks very sharp and clean. There is some unexpected behavior though. With the OVR Metrics tool running, the GPU level is at 5 and usage is at about 60% which is quite high. The app also can't keep 60fps, it runs around 45 fps, although I could run it at 60 fps before switching to the OVR overlay. In addition, there is some very real artifacting that makes it currently unusable. It seems like there is some judder especially when the frame rate is low. There is what to me looks like small brief moments of small bits of the screen that distorts and glitches. These glitches are a few ms long so I can barely see them but they are there. I am also using another OVR Overlay layer for buttons as well. Also the app takes about 15 seconds to start up which is extremely long. Ill report back with any updates.

[robertncoomber]

The overlay layer is not designed to render 360 content. 180 content works however, 360 content produces a thin black line on the back side of the video and there is no way to get rid of it.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale[bot]]

This issue has been automatically closed because it has not had recent activity. If you wish to continue this issue, then please create a new issue as we do not monitor closed issues.

[whatisor]

we need to follow this one: https://developer.oculus.com/documentation/unity/unity-sf-stereo180video/

[lightsailvr]

With the upcoming AVPro 2.0 and the Ultra option - has this feature been added? The difference in quality is astounding when pushing to the OVR Overlay instead of rendering on a sphere. If so, it would be an easy choice to pay to upgrade to 2.0, but I can't find it mentioned in any of the release notes.

[Ste-RH]

This is still on the roadmap, but not as yet included in version 2

[evan-spiegel]

I was able to successfully render a texture from AVPro to an OVR overlay layer. It is rendering a stereo 4k 360 video using Oculus Integration 1.44 and AVPro 1.10.1. I took some demo code from Oculus's Stereo180Video.unity sample scene to properly set up the Overlay layer.

I feel like the video is much sharper on the OVR overlay layer. However, I am noticing some strange color artifacts. Playback on the OVR Overlay layer is much brighter. In addition there seems to be some compression happening on the colors. I am not a compression expert but there seems to be less colors rendering.

I've only spent a couple hours playing with it so there might be something I am missing or set up incorrectly. I am going to continue working with it to try to get it rendering properly.

But being able to assign it to an external texture in theory would produce even better results. I would certainly use it.

Edit: The color is fixed by switching color space to gamma. It is properly rending in stereo and the video looks very sharp and clean. There is some unexpected behavior though. With the OVR Metrics tool running, the GPU level is at 5 and usage is at about 60% which is quite high. The app also can't keep 60fps, it runs around 45 fps, although I could run it at 60 fps before switching to the OVR overlay. In addition, there is some very real artifacting that makes it currently unusable. It seems like there is some judder especially when the frame rate is low. There is what to me looks like small brief moments of small bits of the screen that distorts and glitches. These glitches are a few ms long so I can barely see them but they are there. I am also using another OVR Overlay layer for buttons as well. Also the app takes about 15 seconds to start up which is extremely long. Ill report back with any updates.

Hi there, could you go into more detail as to how you did this? My app is currently rendering 360 video using a sphere and the AVPro Media Player, but I'm trying to render this video to an OVR Overlay. I added an OVR Overlay component to the sphere, matched its settings with that of the half sphere in the demo scene you mentioned, made a copy of Oculus's MoviePlayerSample script and am building off of that, but so far all I've gotten is a light blue sphere covering up the video. The video is still being rendered to the sphere though. I used andriworld's line of code...

ovrOverlay.textures[0] = mediaPlayer.TextureProducer.GetTexture();

...in order to assign the video texture to the overlay. What else did you have to do to get this to work? Thanks!

[evan-spiegel]

Hi @lightsailvr, I got this working by taking the texture from AVPro (as soon as it's not null) and assigning it to OVR Overlay: ovrOverlay.textures[0] = mediaPlayer.TextureProducer.GetTexture();

It works great! What would be even better is if we could assign the pointer to the native android texture, but AVPro doesn't seem to expose it 'ovrOverlay.externalSurfaceObject = ???'

Hi there, I know this was a while ago but can you maybe give some more details as to how you got this working? I found the place in the code for Oculus's MoviePlayerSample script to put your line of code, and that seems to be working in terms of sending/rendering the video texture from AVPro to the OVR overlay layer (the big sphere that gets spawned when I hit Preview In Editor), but I can't seem to be able to see that sphere in play mode. I can see the sphere I already had, but not the bigger one. Any help you can give is much appreciated, thanks!

[robertncoomber]

@evan-spiegel Here is the code I am using to copy the texture from the AVPro video player onto the OVR Overlay. Hope this helps.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using RenderHeads.Media.AVProVideo;

public class AVProOVR : MonoBehaviour
{
    [SerializeField]
    OVROverlay overlay;
    [SerializeField]
    MediaPlayer mediaPlayer;
    private Renderer mediaRenderer = null;

    private void Awake()
    {
        mediaRenderer = GetComponent<Renderer>();

        Rect destRect = new Rect(0, 0, 1, 1);
        overlay.currentOverlayShape = OVROverlay.OverlayShape.Equirect;

        Rect sourceLeft = new Rect(0.0f, 0.0f, 1.0f, 0.5f);
        Rect sourceRight = new Rect(0.0f, 0.5f, 1.0f, 0.5f);
        overlay.SetSrcDestRects(sourceLeft, sourceRight, destRect, destRect);

        overlay.enabled = false;

        overlay.enabled = (overlay.currentOverlayShape != OVROverlay.OverlayShape.Equirect || Application.platform == RuntimePlatform.Android);

        #if UNITY_EDITOR
        overlay.currentOverlayShape = OVROverlay.OverlayShape.Quad;
        overlay.enabled = true;
        #endif
    }

    private void Update()
    {
        var displayTexture = mediaPlayer.TextureProducer.GetTexture() != null ? mediaPlayer.TextureProducer.GetTexture() : Texture2D.blackTexture;

        if (overlay.enabled)
        {
            if (overlay.textures[0] != displayTexture)
            {
                // OVROverlay won't check if the texture changed, so disable to clear old texture
                overlay.enabled = false;

                print("DisplayTexture Render Texture width: " + displayTexture.width + " height: " + displayTexture.height); 

                overlay.textures[0] = displayTexture;

                mediaRenderer.material.mainTexture = displayTexture;
                mediaRenderer.material.SetVector("_SrcRectLeft", overlay.srcRectLeft.ToVector());
                mediaRenderer.material.SetVector("_SrcRectRight", overlay.srcRectRight.ToVector());

                overlay.enabled = true;
            }
        }
        else
        {
            mediaRenderer.material.mainTexture = displayTexture;
            mediaRenderer.material.SetVector("_SrcRectLeft", overlay.srcRectLeft.ToVector());
            mediaRenderer.material.SetVector("_SrcRectRight", overlay.srcRectRight.ToVector());
        }
    }
}

[evan-spiegel]

@robertncoomber Thanks so much for your response, I'll try that!

[LYLeon]

Is there a plan to support OVR Overlay natively? I tried using texture from the player and set it on the Overlay, the texture looks much sharper, but it suffers from the Gamma problem.... I also tried to fix it with a shader that does pow(color, 2.2), it looks ok but in the darker area color becomes pixelated. Please support the OVR Overlay natively!!

[OliverRudoll]

@RenderHeadsSte i bought AVPro because you claim that AVPro can play 8K 360 Video e.g.: on Oculus Quest 2 (it work's) but without a nativ support for the OVR Overlay (or a different Solution that creates more sharpness) the visual quality is below average / e.g. below Oculus TV. AVPro is not made for a Pro / HQ use case - can you change that?

[robertncoomber]

@OliverRudoll AVPro does work well with OVR Overlay. It just doesn't work right out of the box. It's been a while since I've looked at the code but you will need to pass the texture produced from AVPro and pass that along to the OVR Overlay. You can either do this through code or through a render texture.

[AndrewRH]

This feels like something we need to test and add to our documentation... or perhaps create a helper component for to improve integration..

[AndrewRH]

@robertncoomber do you remember whether you did this using the OES texture? Any information will help us to make this a more solid feature.

[OliverRudoll]

@robertncoomber thanks for your reply - can you define "does work well" and on what device? i really need this to work properly with mono 360 - 7200x3600 CBR 100mbps on an Android based Oculus Quest 2 (Mobile not over Link) - i'm testing on a Quest 2 using OES and Exoplayer. @AndrewRH a proper working example scene / code would be great. Unfortunately i don't have time for a research project.

[robertncoomber]

@OliverRudoll "Does work well" = I have an app running on stand alone Quest 2 that plays back a stereo(over/under) 3840*3840 h264 and h265 video exceeding 100 mbps at multiple points. Using the OVR Overlay layer is the highest possible visual sharpness you can get on the quest i.e. what oculus uses for their video playback.

Since it sounds like you have already gotten the 7200*3600 video to playback on the quest (w/o ovr overlay), playback performance shouldn't be an issue.

You will want to be using Exo player, but OES texture doesn't matter if using OVR overlay since you wouldn't be using that material/shader.

That app is no longer maintained and I don't have access to the code but the following code should help you get started. I don't have time to test this on the headset so their may need to be some adjustments but this is essentially how it's done.

using UnityEngine;
using RenderHeads.Media.AVProVideo;

public class AVProToOverlay : MonoBehaviour
{
    // assign these in the editor
    public MediaPlayer mediaplayer;
    public RenderTexture rt;
    public OVROverlay overlay;

    private Texture avProVideoTexture;

    // use this to apply the AVPro texture to a RT
    // put this in a the update method
    // Graphics.Blit is computationally expensive, may not be efficient enough for very large videos
    void AVProToRenderTexture()
    {
        Texture avProVideoTexture = mediaplayer.TextureProducer.GetTexture();
        Graphics.Blit(avProVideoTexture, rt);
    }

    // assigns the texture straight to the ovrOverlay texture properties 
    // this may work in only the start method but it might need to be in the update method
    void AVProToOVROverlay()
    {
        Texture avProVideoTexture = mediaplayer.TextureProducer.GetTexture();

        overlay.textures[0] = avProVideoTexture;
        overlay.textures[1] = avProVideoTexture;
    }
}

Use the Media Players TextureProducer to get the texture and pass that texture along to the OVR Overlay. Both options work, I would recommend getting the AVProToOVROverlay() up and running as it will be more efficient.

Hopefully this is enough to get you up and running. I would recommend checking out the OVR Overlay sample in the Oculus package if you want to know more about how the OVR Overlay works.

[robertncoomber]

@robertncoomber do you remember whether you did this using the OES texture? Any information will help us to make this a more solid feature.

@AndrewRH Since I was using the OVR Overlay, I don't believe I used the OES texture Material/Shader since the OVR Overlay handles the rendering w/o a material.

Since this is something I had to take time to figure out and program when building my app, it would have been helpful to have this built into the AVPro package. I do think that something like this wouldn't be too difficult for the AVPro plugin.

I would prefer it work similarly to the Apply To Material / Apply to mesh. Have it be "Apply to OVROverlay" and take the OVROverlay as a public variable, rather than a mesh or material. There wouldn't need to be any OVR Overlay setup or configuration since that is typically in the OVR Overlay component in the editor. I believe I actually used the apply to mesh and material components' source code to figure out how to build my own "Apply to OVROverlay" component.

I don't have any Quest projects at the moment, so getting that set up again will be a process but I would be happy to help if there is anything else I can do to make this feature happen.

[Ste-RH]

We are certainly keen to see what we can provide to support the Oculus OVR Overlay and indeed did look at it a while back. If someone could provide a working OVR example project then this will help us and hopefully reduce the time to inclusion. Or maybe there is a fitting example scene in the Oculus SDK? Such a demo project can be emailed over to us at unitysupport@renderheads.com with the subject #63. Otherwise, I will put it on the list and hope we can get to it soon.

[colinhallett]

Hi - I was curious if there's been any movement on this?

I've managed to get things working with OVR Overlay + AVPro v3, and using the texture from mediaPlayer.TextureProducer.GetTexture(); and assigning it to the overlay's texture. However am running into significant framerate issues - essentially the same ones outlined here - https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/63#issuecomment-575814163

In the Meta Quest samples, they are using this property - 'ovrOverlay.externalSurfaceObject = ` see https://github.com/oculus-samples/Unity-StarterSamples/blob/f2d296abe4188b3e43146427c923316e846006e3/Assets/StarterSamples/Core/Video/Scripts/MoviePlayerSample.cs#L236