Doesn't look 3D in VR (Oculus Quest 2)

[bittu97]

The model doesn't look 3D if we build it in Oculus or in VR.

[bittu97]

The model doesn't look 3D if we build it in Oculus or in VR.

Solution (By Matias Lavik): Just change stereo render mode to multi-pas, in Unity. This works fine.

[mlavik1]

Thanks for adding it here as well :) I'll keep the task open for now, since the actual issue has not been solved (should work without multi-pass).

[karstennoe]

Hi Matias. Have you fixed the shaders for single pass stereo? Otherwise I have made the changes locally in my own repo. It works (tested on Varjo Aero). Might as well be put back in the main repo?

[mlavik1]

Hi @karstennoe , thanks for asking! I'm no longer working in the VR industry, and don't have a headset. My friend will borrow me one after new year, so then I should finally be able to look at some of the VR related issues. So no, I have not been able to fix this. Feel free to make a PR! That would be very much appreciated!

[karstennoe]

Hi again.

What I have done is simply to enable stereo support in the shaders:

However, I might have jumped the gun a bit on this, as I experience some issues with composition of volume rendering and polygon mesh rendering. When running e.g. a VR glove through the volume it is clear that left and right eye views don't aggree on where the volume object and mesh object intersect - in my view this indicates that the two rendering modes don't completely agree on model, view and/or projection matrices for at least one of the eyes (the effect is too large to merely be numerical issues with depth estimation in the volume rendering IMO).

In VR this can be quite disturbing - especially when getting close to the object, so I better investigate further before creating a PR.

[mlavik1]

Thanks for sharing! I'll see if I can also look into it a bit when I get a VR headset.

Do you notice this issue when using multi-pass rendering also?

[karstennoe]

Hi again

Thanks for responding.

Do you notice this issue when using multi-pass rendering also?

After fighting all morning with getting the Varjo base software working again after it mysteriously refused to work (happens once in a while for me), I finally managed to test that out. And it does indeed seem like the issue is not present when using the multi-pass rendering. However the rendering frame rate is also significantly lower in multi-pass mode than single-pass instanced mode.

I took the screen shot attached (or rather I had to yell for someone to come and click the screenshot button ;-) ). It looks to me like the volume renderings are displaced with respect to each other with what could be the eye spacing.

[karstennoe]

Hi again!

Good news! I managed to fix it.

I had missed adding the magic spell "UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX(i)" in the fragment shader. I think what happened was that the built in function ObjSpaceViewDir was incorrectly computing object space view directions for the second eye as if has been rendered from the first eye.

Sorry for wasting your time thinking over this!

[mlavik1]

Nice, so it's working fine with that change? Thanks a lot for looking into this :)

I made a PR with your suggested changes. Would you mind trying it out for me? See #148 148

"or rather I had to yell for someone to come and click the screenshot button" -> Oh yes I've done that a lot when I worked with VR haha :joy:

[karstennoe]

Hi again

Thanks a lot for looking into this :)

Instead I should thank you for this repo. When starting my post doc I had imagined having to write the volume renderer myself, so you saved me a lot of work ;-)

Nice, so it's working fine with that change?

Yes, I think it looks as it should now (in my repo). There are some issues when using the Varjo XR backend (as opposed to the OpenVR or OpenXR ones) with foveated rendering and motion smoothing. I think it is related to the Varjo compositor having problems with motion smoothing all the semitransparent volume fragments here and there (and where the reported depth map is only corresponds to the end of ray) - so I think this could be expected.

I will check the PR out and test it.

By the way: how do you suggest using the UnityVolumeRendering code in connection with another unity project? Wouldn't it make sense to make it a "real unity package" (with a package.json file) that could be imported from the package manager (from disk or git url)? Then it would be easier for someone like me to align with the main repo and push fixes or perhaps even extra features. I might experiment with what creating a package would entail when testing the PR.

[mlavik1]

Really glad to hear that! :)

Hmm.. yes that could be! What kind of issues do you end up seeing when using motion smoothing? And thanks! Please let me know when you've had time to test the PR, so I can merge it.

Hmm.. good question! Currently, my recommended ways to use it would be to: A. Add it as a git submodule (I personally like this way). B. Use the unity package builds C. Use the Asset store version. It's a pain to update the asset store version though (have finally set up automatic build scripts for that now though!).

I don't have much experience with "real" unity packages though (using package.json), so I'll need to read up on that a bit first, but that's probably something I should do :) There's one thing I'm not sure about though.. I'd like people to be able to simply pull my repo, open it as a project and run the sample scene. Is there a way to achieve both? I suppose I could keep the repository as a project (like now) and place the actual package in a sub-folder, and have the outer project reference it (as a local package). But I suppose "Add package form Git URL" wouldn't work then, since the package is placed in a subfolder of my repo.. Am I missing something maybe? :sweat_smile:

[karstennoe]

Hi Matias

The PR branch works in single pass instanced mode on both the OpenVR shipped with SteamVR, OpenXR (on Arjo) as well as the Arjo Mode. So I think it is safe to merge.

Here is an image of the artifacts I mentioned (look at the distortion of the ground plane):

They can only be seen when moving the HMD in Varjo mode or OpenXR with depth output enabled (so that varjo can do motion compensation in its compositor). So I am quite convinced it is an inherent issue in the motion compensation approach (for transparent objects) rather than something to be handled in your repo. (It is related to what depth is reported for something semitransparent which is not well defined IMO).

Whether or not the foveated rendering also leads to issues is harder for me to determine.

A note though: In this case OpenXR has a significantly lower performance than the two other modes (and OpenVR has a lower resolution than Varjo). So if somebody complains about performance in OpenXR mode, perhaps suggest to try OpenVR?

[mlavik1]

Nice, thanks a lot for testing! I'll just go ahead and merge the PR then :) Looks like I'll get a VR headset to test with next week btw, so it should be easier for me to look at VR related issues then! I still haven't tried with a Varjo headset, even though I worked there a while ago :sweat_smile: Nice to know that it's working there too!

I'll add your suggestion to the new FAQ as well. Do you think it applies to other VR headsets as well?

[karstennoe]

Sure!

I have no other headsets to test on, so I couldn't be sure! It may only be a Varjo thing. It sounds like more advanced stuff is happening in varjos OpenXR backend than in the OpenVR one, and indeed the difference may simply be down to the OpenXR supporting a higher resolution than OpenVR on Varjo - and thus lower frame rates. However as you know getting under a certain performance threshold in VR can be quite nauseating!

By the way, I ended up taking the git submodule approach - I think that works fine!

[mlavik1]

Hmm, yes that could be! I'll probably find out on Thursday then, if I get my VR headset :D (a Quest)

By the way, if you're struggling with low framerates, have you tried using DLSS? I've never used it with this project, but I can imagine it will give a nice performance boost since the bottleneck without doubt is the pixel shader. It requires HDRP though - I'm not sure if you'll need to modify the shaders to get them to work (with VR), or if the auto-upgrade tool fixes it.

Ok, nice to hear!