Android Oculus Quest URP + OES + Stereo eye rendering issue

[JuanObiJuan]

Describe the issue When using Android OES, the result is glitchy/stuttering. When not using the video is smooth.

I tried already without the whole texture setup just ApplyToMaterials and the result is the same. Just on OES

My Setup:

• Unity version: 2020LTS 3.4.3f1
• URP
• Stetero Render Mode: Multipass rendering
• Update Multipass Stereo script attached to the Ceneter Eye Camera
• Media Player with resolve To RenderTexture
• External Texture matching the size of Cones-360Stereo-2K30-H264 2880x2880
• AVPro Video: v2.6.7 (plugin v2.6.7f1-ultra)
• Operating system version: Android
• Device model: Oculus Quest 2
• Video specs (resolution, frame-rate, codec, file size): Cones-360Stereo-2K30-H264.mp4

• 

To Reproduce

1. Enable OES
2. Build APK
3. Try on device

Logs There are no errors

Gif Screen capture

[makuchan]

(Japanese to English Google Translate)

I have the same symptoms. I'm using it for the application that will be delivered today, so please respond as soon as possible! This phenomenon is from version 2.6.7. No problem with version 2.6.6. You can also downgrade to 2.6.6 in Unity 2021.3.16f environment. Please tell me what to do.

[Ste-RH]

This appears to be a right/left eye flicker. You can confirm this by enabling 'Stereo Debug Tinting' in the shader. You then see a red green flicker in both eyes.

This is odd because we fixed a bunch of issues around this problem last year and it was tested quite extensively.

The problem I think is actually linked to Unity incorrectly setting up either UNITY_MATRIX_I_V or _WorldSpaceCameraPos in each eye pass of the shader - see IsStereoEyeLeft() in AVProVideo.cginc

Can you switch to Multiview?

[Ste-RH]

(Japanese to English Google Translate)

I have the same symptoms. I'm using it for the application that will be delivered today, so please respond as soon as possible! This phenomenon is from version 2.6.7. No problem with version 2.6.6. You can also downgrade to 2.6.6 in Unity 2021.3.16f environment. Please tell me what to do.

I have tested 2.6.6 here and the issue exists in this version also - as the shaders have not changed for many months since we thought we had resolved this issue.

[Ste-RH]

After doing some testing, looks like this is an issue in Unity 2021 and onwards. The issue is not there in Unity 2020 and lower. We will look for a fix, but so far nothing obvious has presented itself.

[JuanObiJuan]

Hi @Ste-RH The issue is also present on 2020 LTS version

[Ste-RH]

Which version of 2020 did you test in @JuanObiJuan? 2020.3.43? Will check that here now...

[makuchan]

I had no problems with 2.6.6, so please investigate further. Since the version cannot be dropped to Unity 2020, please modify it so that it works with the latest version of Unity 2021 LTS.

[Ste-RH]

I suggest you check again @makuchan. The issue happens because the variables that the stereo shaders use to determine which eye is being rendered in each pass are simply not available in the outlined versions of Unity. There were no changes to these shaders between AVPro Video v2.6.6 and v2.6.7, and so the issue would have been present in v2.6.6.

We are investigating, but as things stand there is no fix available as it has come about in changes in Unity itself. I would recommend you change to Multiview if possible.

[JuanObiJuan]

Hi @Ste-RH all the details of the version and settings are on the top of the issue thread and also the gif with the issue, so you can see that the issue is there. Copy/paste again:

Unity version: 2020LTS 3.4.3f1 Stetero Render Mode: Multipass rendering Update Multipass Stereo script attached to the Ceneter Eye Camera Media Player with resolve To RenderTexture External Texture matching the size of Cones-360Stereo-2K30-H264 2880x2880 AVPro Video: v2.6.7 (plugin v2.6.7f1-ultra) Operating system version: Android Device model: Oculus Quest 2 Video specs (resolution, frame-rate, codec, file size): Cones-360Stereo-2K30-H264.mp4

Im stucked on 2020 LTS and I can not go to other versions futher than that one. Unity is not fixing on 2021LTS some performance issues for Oculus Quest so I can not upgrade my project.

With your recommendation to change to multiview you mean not to use android OES? The main reason for me to use OES (and also why I bought the plugin) is to be able to play highres 360 stereo videos. Without oes the headset is not performing good enough and I could use then the Unity native video player.

[Ste-RH]

2020LTS 3.4.3f1 <- no such version exists

https://unity.com/releases/editor/archive

These do:

2020.3.4 2020.3.43

...but not 2020LTS 3.4.3f1

Multiview + OES should work, maybe give it a try?

[JuanObiJuan]

https://unity.com/releases/editor/qa/lts-releases

The version exists. This is the last 2020LTS release, I think they are the same listed under non LTS version because the were releases the same day 15 December but Im not sure.

I will try multiview.

[Ste-RH]

Yeah, so you mean 2020.3.43f1 not 2020LTS 3.4.3f1

Either way, it appears that the issue seems to exist in 2020, 2021 and 2022. I will investigate what can be done to address it, but as stated...this looks like a failing all Unity with the following variables:

• unity_StereoEyeIndex
• UNITY_MATRIX_I_V
• _WorldSpaceCameraPos

[makuchan]

Unity 2021.3.16f1 MetaQuest2 Android 10.0 API level 29 Mono ARMv7

XR Interaction Toolkit 2.2.0 XR Plug-in Management 4.3.1>Oculus

[Stereo Rendering Mode] Multiview -> Multi Pass

Symptoms that occurred for 1 to 2 seconds were reduced to about 5 seconds.

[Phase Sync] OFF -> ON

Slightly reduced?

---

Android 10.0 API level 29 IL2CPP ARM64

XR Plug-in Management 4.3.1>Oculus

Stereo Rendering Mode: Multi Pass Low Overhead: OFF Optimize - Subsampled: ON Foveated: Fixed Enable TrackingOrigin: ON Quest2: ON Quest Pro: ON

The occurrence was suppressed at about once every 12 seconds.

[makuchan]

I checked again.

Unity 2021.3.16f1 MetaQuest2 Android 10.0 API level 29 IL2CPP ARM64 Stereo Rendering Mode: Multiview

Checked the behavior of the problem with the product version AVPro Video - Core Android Edition 2.6.7. Delete AVProVideo in the project, download the 2.6.6 trial version from Github and import the package. https://github.com/RenderHeads/UnityPlugin-AVProVideo/releases/tag/2.6.6

I have confirmed that there is no problem in operation.

Stereo 2880x2880 H264 AVC1 yuv420p 30fps : Cones-360Stereo-2K30-H264.mp4 (Demos) Stereo 5760x5760 HEVC (Main10) yuv420p10le 59.94fps AAC Stereo 4096x4096 HEVC (Main10) yuv420p10le 59.94fps AAC

How can I download the retail version of AVPro Video - Core Android Edition 2.6.6 again until it is fixed?

[Ste-RH]

You say you are using Multiview in the last test that did not show the issue. That is the reason it worked, not because you are using v2.6.6. Switching to Multiview and using 2.6.7 should also be good.

For info... Unfortunately there is no provision in the Unity Asset Store to get older versions.

All the above said. We may have a fix for the issue. I am in the middle of testing it and will post up the shader once we verify it works (still need to test SRP + OES but my Quest 2 ran out of battery!)

[Ste-RH]

Ok, this is not the final fix, as SPR + OES does not work with this change. But you can test it to verify it addresses your issue. The fix will likely not be incorporated until v2.7.1 (v2.7.0 is dues for release this week).

Extract the following file into '\Assets\AVProVideo\Runtime\Shaders\' and overwrite the existing file:

[Ste-RH]

...and here is the full fix. Extract into your Assets folder and overwrite all files.

Do let me know if this fixes the issue for you @JuanObiJuan @makuchan

AVProVideo.zip

[makuchan]

@Ste-RH Thank you for your correction.

I tried both Multi Pass/Multiview, but no improvement. I checked by overwriting the Runtime folder after importing AVPro Video - Core Android Edition 2.6.7. The result was the same behavior as the current 2.6.7.

my environment is

Unity 2021.3.16f1 Windows10 MetaQuest2

URP Android 10.0 API level 29 IL2CPP ARM64

[Ste-RH]

Everything works here just as I would expect. I have tried every option I can think of to break it, but cannot.

Can you send over a project that reproduces it @makuchan ? Email it over to unitysupport@renderheads.com

[makuchan]

@Ste-RH I sent you an email. Since the URL is listed, it may be misidentified as spam. please confirm.

Also, I confirmed the operation with the released 2.7.0, but it was the same behavior as 2.6.7.

[Ste-RH]

No fix for this issue made it into v2.7.0 (as denoted by no mention in the release notes).

Just picked up your email/project. Will get back to you shortly after investigation.

[Ste-RH]

@makuchan in the sample scene you sent...it is not set to OES. I assume I need to be changing the shader/MediaPlayer to OES to reproduce? Also I need to change Multiview to Multi Pass?

[makuchan]

@Ste-RH

Is this the only setting?

There is no problem with multipath if it is displayed in stereoscopic view on Quest.

[Ste-RH]

That and you need to change the shader to the OES version.

Here is what I have changed in your project:

With these changes, the issue does not happen still. Please advise on how I might reproduce it with the project you sent.

[Ste-RH]

There is no problem with multipath if it is displayed in stereoscopic view on Quest.

'multipath' is not an option. Only Multiview and Multi Pass exist. Which one is the issue? (in your first post you say 'Multipass' is the problem)

[Ste-RH]

With the issue happening, can you enable this option on the shader:

You should see a green tint in the left eye, and a red tint in the right eye. Do these tints change/flicker?

[makuchan]

@Ste-RH Thank you! The problem appears to have been resolved. But either the sphere projecting the video or the position of the camera is wrong, so I'm trying to figure out what the problem is. how are you resolved?

[Ste-RH]

So the fix I put forward does actually fix the issue you were seeing?

But either the sphere projecting the video or the position of the camera is wrong, so I'm trying to figure out what the problem is.

Is this a new issue? Can it be seen in the project you sent?

[makuchan]

Since I updated to 2.7.0, I will revert to 2.6.7 and rebuild to check. please wait a little bit.

[makuchan]

About shaders

[AVProVideo/VR/InsideSphere Unlit (stereo+color) - Android OES ONLY] and [AVProVideo/VR/InsideSphere Unlit (stereo+fog)]

It looks like the angle of view calculation for the projected video is different. I haven't followed the shader code yet, but it looks like there is a difference between EquiRect180 and EquiRect360.

The camera position problem was avoided by setting the Tracking Type of the Tracked Pose Driver (Input System) to Rotation Only.

[Ste-RH]

There is a texture transform that get applied in the OES shader that is not in the none-OES. However, I believe that same transform is applied at android SDK level in the none-OES render path.

You could try removing that transform. The line you are looking for it:

texVal.xy = (_TextureMatrix * vec4(texVal.x, texVal.y, 0.0, 1.0)).xy;

...and...

uv.xy = (_TextureMatrix * vec4(uv.x, uv.y, 0.0, 1.0)).xy;

...depending if you are using the HIGH_QUALITY flag.

it looks like there is a difference between EquiRect180 and EquiRect360

Yes, one is for 180 videos and the other 360. If you are squashing a 360 into a 180 render, it is not going to look correct...and the other way around also won't look correct.

[makuchan]

@Ste-RH Problem solved! Thank you for supporting me for so long.

[Ste-RH]

@Ste-RH Problem solved! Thank you for supporting me for so long.

Which problem? Both? If so, what was the solution to the projection issue?

[makuchan]

yes. Both. The camera angle of view of Stereo VR360 probably does not seem to be a problem, but we will check the scaling etc. with another VR360/VR180 video.

[makuchan]

Stereo VR360 5760x5760 HEVC (Main10) yuv420p10le 59.94fps AAC I was able to confirm the playback was normal.

[Ste-RH]

Super. We will get the fixes in for the next release (v2.7.1)

[Chris-RH]

The latest version has been released, please let us know if this has not fixed the issue for you.