search

mrdoob / three.js

JavaScript 3D Library.
https://threejs.org/
MIT License
102.97k stars 35.4k forks source link

setProjectionFromUnion makes incorrect assumptions #16766

Open cabanier opened 5 years ago

cabanier commented 5 years ago

The comment from setProjectionFromUnion states:

Assumes 2 cameras that are parallel and share an X-axis

For Magic Leap devices, that is not the case and because of this, this function calculates incorrect values which results in the scene not rendering correctly. It's observable by virtually pinning content to a wall or whiteboard and then walking away. You will then see that the content will rotate or move from its location.

WebXR specifies that projection matrices are not supposed to be decomposed:

It is strongly recommended that applications use this matrix without modification or decomposition. Failure to use the provided projection matrices when rendering may cause the presented frame to be distorted or badly aligned, resulting in varying degrees of user discomfort.

cabanier commented 5 years ago

There is a way to calculate the half-angles from the projection matrix when the cameras are not parallel, but maybe it would be better if the browser provides these values since it's quite complicated.

@mrdoob, why is three.js calculating another projection matrix?

cabanier commented 5 years ago

We looked the code over and we understand what it's doing now. We're planning on updating the algorithm to work with any set of projection matrices.

mrdoob commented 5 years ago

/cc @jsantell

jsantell commented 5 years ago

Any assumptions were for optimization reasons; if that doesn't hold true on magic leap, then we probably need to decompose both projection matrices -- decomposing the projection seems OK as this is only used for frustum culling, yeah?

What assumptions break on magic leap, are the cameras not parallel?

It's observable by virtually pinning content to a wall or whiteboard and then walking away. You will then see that the content will rotate or move from its location.

Hm this should only affect culling..

cabanier commented 5 years ago

Any assumptions were for optimization reasons; if that doesn't hold true on magic leap, then we probably need to decompose both projection matrices -- decomposing the projection seems OK as this is only used for frustum culling, yeah?

Yes. The link to the stackoverflow article proposes a more generic solution. We're using that to create another algorithm.

What assumptions break on magic leap, are the cameras not parallel?

Correct. Because of eye tracking, the projection matrices will contain a unique ipd, fov values and an additional rotation component.

It's observable by virtually pinning content to a wall or whiteboard and then walking away. You will then see that the content will rotate or move from its location.

Hm this should only affect culling..

You're correct. This is caused by something else. We're investigating if this is a bug on our side or three.js. Since other sites look fine, we suspect it's three.js but we won't file an issue until we're sure.