OES Rendering Path on Oculus Quest is rendering in a Sphere below the Camera instead of around the Camera

[LB-Till]

Issue description When trying to use OES Path with ApplyToMesh on Oculus Quest, the video is rendering on a small sphere below the camera instead of a sphere around the player.

Also when choosing Multiview, the one eye gets an offset.

My Setup

• Unity version: 2021.3.0f1 (LTS)
• AVPro Video version (number and edition (trial/core/ultra/enterprise)): AVPro Video Android Core 2.5.2
• Operating system version: Android - Oculus Quest (41.0.0.xxx)
• Device model: Oculus Quest
• Video specs (resolution, frame-rate, codec, file size): Cones-360Mono-4K30-H264 from Demo Videos
• URP 12.1.6
• XR Plugin Management 4.2.1
• Oculus XR Plugin 3.0.2

To Reproduce

1. Create an empty project,
2. Import XR Plugin Management 4.2.1
3. Import Oculus XR Plugin 3.0.2
4. Import Universal Render Pipeline 12.1.6
5. Create and Add URP Pipeline Asset under Graphic Settings
6. import AVPro Video Core Android Edition 2.5.2
7. Set in Project Settings: 7a. Graphics API to "Open GLES3" 7b. Color Space Linear
8. Create new Scene, add Tracked Pose Driver to Main Camera
9. Add AVPro Video - Media Player 9a. Add Video 9b. In Platform Specific Settings - Android: Set Use OES Rendering to enabled
10. Copy "Sphere-ApplyToMesh" from Demo_360Stereo scene. (Should have 100,100,100 Scaling and Pos at 0,0,0) 10a. Fill in Media Player 10b. Swap Shader for the Demo-360 Material to InsideSphere Unlit - Android OES ONLY
11. Build Scene and Run.

Logs

Screenshots Screenshots are ODH captures, so ignore the double eye effect.

Screenshot with multiview

Videos

[Ste-RH]

I only have a Quest 2 to hand - which I wouldn't think should make a difference.

Following your steps exactly, I do not get what you are seeing. I get a perfectly rendered sphere around the camera playing back the video as expected. This is in both Multi Pass and Multiview.

Can you email over the project you created so I can see what might be different? If so, send it to unitysupport@renderheads.com

[LB-Till]

Ok, just confirmed that i do have the same issue on oculus quest 2.

At this point, I assume i forgot a checkbox somewhere or used a weird setting. I've sent the project to you guys. Thanks in advance.

[Ste-RH]

Thanks for testing/confirming re Quest 2, and sending over a project. Have requested access to the download.

[Ste-RH]

The difference is quote simple.

You are not using the Mono sample video, you are using the Stereo sample video. This is a top/bottom stereo packed. If I change it to the Mono video, everything is rendered fine. If I leave it as the stereo video and change the packing to top/bottom, everything is fine.

[LB-Till]

ok. I've just tested it. Set stereo packing to TopBottom, kept stereo video. Same Result as before: With sphere at bottom and double image when using multiview. Also tried keeping stereo at none and taking the mono video. Same result.

[LB-Till]

Is there a shader to use the OES path on the skybox instead of the sphere? Might be an option if that is working.

[Ste-RH]

Multi Pass, Multiview, both OES and none-OES all behave exactly as they should do here.

The only other thing I did was zero the camera transform position...but that shouldn't do anything.

No, there is no skybox OES shader. Only the special AVPro Video ones.

[Ste-RH]

[Ste-RH]

Anything to report @LB-Till ?

[LB-Till]

Sorry for the late reply. We've had a bank holiday and a long weekend over here last thursday.

I've just created a new build. Reset camera and checked all the settings. Issue persists.

[LB-Till]

Ok, i think i found a quick fix. There seems to be an issue with the conversion of the model itself. I created a sphere in blender and it was tiny in the build. Once i increased the scale factor in the import settings, it has the increased size. So i assume, for some reason my import is not working correctly.

With the custom sphere i do have some clipping with the corners though, which is probably because my sphere isn't as smooth as yours?

EDIT: Multiview is still broken for me, rendering two separate images that overlap. (Only the rendering sphere though, a default cube inside the sphere is rendering fine)

[Chris-RH]

Yeah, in blender you need to make sure you've used the modifier "subdivision surface" and increase the levels. Also Object menu -> shade smooth.

Is it just your own sphere that is not rendering properly still or is it the default demoscene one?

[LB-Till]

Neither sphere is rendering correct in multiview (separate images for each eye). Multiview itself is working correctly though. Default sphere is still rendering too small. My own sphere is rendering fine, except for the smoothing issue (which can be probably solved easily, by increasing the polycount).

I tried all different combinations of shader and media player settings. And using both mono and stereo videos. To me it looks like an issue in the shader or shader execution, but i can't get my head around it. What or why its not working properly.

In the screenshots below you can see the difference between left and right eye. Both screenshots using the custom sphere. The cube is fine for both eyes, the video is separated. The yellow spot is a near plane clipping issue, that requires some finetuning.

[Ste-RH]

Can you email over another project showing the above so we can take a look?

[LB-Till]

sure, but its still the same barebone project, just swapped out the video and the sphere.

[LB-Till]

sent it. Im really curious if you can reproduce it or not. Because if not, there must be something incredibly stupid im missing.

[Ste-RH]

Thanks for sending over the project. I just want to make sure we are looking at the same baseline.

Building/running without any changes at all on a Quest 2...and grabbing a video via Android Studio, this is what I get:

https://user-images.githubusercontent.com/46531149/175299205-3bcb1309-e8c5-49c6-bb56-17116769d874.mp4

[LB-Till]

that looks pefectly fine to me. No shifted image. Just the near plane glitch, which we can ignore. what the hell is going on in my project

[Ste-RH]

that looks perfectly fine to me. No shifted image.

That is what I thought

[LB-Till]

ok, i will do a completely new setup on a different computer. And see if i get the same issue there.

[LB-Till]

Tested on a different computer: Importing the project i sent you & building without changing anything: Exactly same issues as before Setting up a clean & new project fixed the "tiny sphere" issue with the demo sphere. But im still having the shifted image issue. So at least now i can confirm, that the sphere scale and the multiview issue were separate issues.
I also tested the apks on a different device (quest 2) to rule out its a device issue.

I will continue to investigate the issue, but im absolutely confused whats happening here.

[LB-Till]

My findings so far: Using the URP Pipeline Asset in Graphic Settings leads to the "tiny sphere". Removing it, displays the sphere in the regular (working) size.

[LB-Till]

I recently tested this on my home computer with a complete fresh unity setup (2021.3.5f1 LTS). Fresh project, only downloading urp, xr plugin management, oculus plugin and avpro video core android.

I created a URP Pipeline Asset and added it to Graphics and Quality in the Settings. I disabled Auto Graphics API and kicked out Vulkan and then switched to linear color space. In XR Plugin Management i enabled Oculus and in the Oculus Settings i switched to multiview and turned everything off except for target devices.

Using the Demo 360 Stereo Scene:

• in the media player swapped out the Reference to the mono video
• added a tracked pose driver to the main camera
• removed the LookAround360 and Update Multi-Pass Stereo scripts from the main camera
• (swapped out the shader on ApplyToMesh to Inside Sphere Unlit Android OES Only)
• (enabled Use OES Rendering)

It works absolutely fine until i enable oes rendering and use the OES shader. Then the sphere gets rendered scaled down and distorted image in the right eye.

I am completely at my wits end here.

[Ste-RH]

I followed your instructions almost to the letter. The only difference was I made a new URP Global Settings before I made a URP Asset (see screen shot). I get the following output.

https://user-images.githubusercontent.com/46531149/176488588-278fbeeb-b45d-455b-8b44-35418f24df10.mp4

Looks exactly as I would expect (yes, this is OES).

[LB-Till]

thanks, i'll give it another try.

[LB-Till]

unfortunately its still the same. Im pretty sure the URP Global Settings gets generated automatically when assigning the URP Asset to Graphics though.

Could there be an issue during the build process maybe? Are you using the Android SDK shipped with Unity? Or could it be some sort of issue with a unity package that may gets autoloaded for my projects? This all makes absolutely no sense 😕

Also what i've noted, that after building, unity has done some changes with the x86 .so files in the android folder even though ARM64 is selected as target architecture. I have no idea if that is normal, i just found it odd.

[Ste-RH]

Are you using the Android SDK shipped with Unity?

I used a complete stock fresh install of the Unity version you said you were using.

The x86 is metafile differences. Interesting as I would actually expect to see no changes, or more than just x86. That is something I will look into...but it will be nothing to do with the odd things you are seeing. You could delete the x86 and x86_64 folders and it will (should!) have no impact/difference.

[Ste-RH]

Your architecture screen grab is interesting. Are you building with IL2CPP? By default Unity sets the Mono scripting backend. Only by changing it to IL2CPP can I then set ARM64. As you never stated a scripting backend change I have always left it on Mono. Will try IL2CPP now...

[Ste-RH]

Again, exactly as it should be.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Chris-RH]

Are you still having issues with this @LB-Till ?

[LB-Till]

hey, we were not able to resolve the problem. For now we are just not using the OES render path.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Ste-RH]

@LB-Till We are pretty sure the issue you were seeing is the same as #1342 and should be addressed in the next release.

[LB-Till]

Great to hear! Thanks for the heads up!

[JuanObiJuan]

This is still present in current version.

My setup is basically (for Oculus Quest 2) the scene included in the demo_360 replacing the camera for the Oculus Quest Camera Rig prefab included in the oculus integration package:

Stereo Rendering Mode: Multipass settings on Android

Media player with OES activated

Material in the sphere: Demo-360 with the shader "AVPro/VR/InsideSphere Unlit Transparent (stereo+color+alpha) - Android OES ONLY"

AVPro Video: v2.6.7 (plugin v2.6.7f1-ultra)

Target Platform: Android

Unity: v2020.3.43f1

Oculus Camera Rig with Update Multipass Stereo script attached to the center eye Anchor (Camera)

Render URP

Playing video "Cones-360Stereo-2K30-H264.mp4"

I tried just before installing the URP packages and the video looks all right.

The same scene, but installing URP package (and setting URP assets, quality, etc on the project), the final reuslt change to the camera on the top of the sphere like in the pictures attached. Doesnt matter how big you make the sphere there is no change.

The same scene but returning to multiview, disabling Android OES and using shader "AVProVideo-VR-InsideSphere-Transparent" everything comes back to how it should and working as expected.

[Ste-RH]

I wonder if it is linked to #1416. Can you grab the early-test-fix I just posted into that issue and give it a test @JuanObiJuan ?

[JuanObiJuan]

Hi @Ste-RH. Same settings with the updated shaders and script from https://github.com/RenderHeads/UnityPlugin-AVProVideo/issues/1416#issuecomment-1372546804 and the result is the same.

Camera is still on the top of the sphere.

[Ste-RH]

Can you send over a project that reproduces this? As usual, to unitysupport@renderheads.com

[JuanObiJuan]

done

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[Ste-RH]

I think this is the same issue as seen in #1613 . There is a fix posed in there that might be worth trying @LB-Till @JuanObiJuan

[Chris-RH]

Have you had any luck with that @JuanObiJuan ?

[Chris-RH]

@JuanObiJuan @LB-Till Any luck with that fix?

[Ste-RH]

Closed due to lack of response.