search

mrdoob / three.js

JavaScript 3D Library.
https://threejs.org/
MIT License
101.58k stars 35.29k forks source link

WebXR Depth Sensing: re-architect how occlusion is done #28877

Open cabanier opened 1 month ago

cabanier commented 1 month ago

Description

Support for WebXR Depth Sensing was added to three after it was implemented by the Quest browser. However, the implementation (and spec) didn't account for offsets between the fov of the cameras and the display. It also didn't do feathering of the depth buffer and the current approach of wholesale copying the depth buffer results in flickering of the scene.

Solution

The correct way to do depth sensing is to sample from the depth buffer (while also accounting for the fov differences) in the material. For unity, the Quest browser created a bunch of custom material changes. I know that this is a hard ask for three but I was wondering if there was a way to get the same behavior without having to change the shader logic in three.

Alternatives

We'd be happy to learn if there are other approaches that might not be as invasive. Our team calls their approach "shadow map". Could we repurpose three's shadowmap logic?

Additional context

One of our engineers that works on depth sensing is going to provide more information.

cabanier commented 1 month ago

cc @mrdoob

cabanier commented 1 month ago

I believe this also affects Android. Is this right, @alcooper91?

alcooper91 commented 1 month ago

I'm doing the reprojection for Android internally. I don't think we settled on a new API shape to expose some of these differences yet either?

cabanier commented 1 month ago

I'm doing the reprojection for Android internally. I don't think we settled on a new API shape to expose some of these differences yet either?

We have not. Hopefully we can discuss it next week.