search

meta-quest / immersive-web-emulator

Browser extension that emulates Meta Quest devices for WebXR development. Lead: Felix Zhang (fe1ix@meta.com)
https://developer.oculus.com/blog/webxr-development-immersive-web-emulator/
MIT License
310 stars 35 forks source link

Planes and meshes are rendered in three.js scene #61

Open marwie opened 1 month ago

marwie commented 1 month ago

Hello,

I just discovered that the emulator is also capable of simulating planes and meshes (which is great!)

However it's very confusing and irritating that the emulator also renders in the threejs scene.

I would expect these planes to not be injected/rendered in my scene unless my app decides to visualize them.

https://threejs.org/examples/?q=xr#webxr_ar_plane_detection

image

image

marwie commented 1 month ago

Additionally it seems that there is a mismatch between the objects rendered by the emulator and the mesh data (the scene generates occluder meshes for detected planes/meshes)

image

felixtrz commented 1 month ago

I understand the confusion, but this feature is supposed to simulate how passthrough works in headset - a video layer showing your physical surroundings that will always be in the background of the scene you are rendering. Similarly for IWE, the injected layer renders the emulated physical environment (planes, meshes, etc.) and they will always be under your scene rendering.

felixtrz commented 1 month ago

Regarding the misalignment of the mesh in your threejs scene and the emulated passthrough layer, it's most likely an HTML issue - the passthrough layer is a separate canvas that sits below your threejs canvas, and they are both supposed to stretch to 100vw by 100vh, which should happen by default (if you can provide me with a test link I can help verify whether this is the case). Another possibility is if you resized the window after entering XR - the background canvas will resize responsively, but your threejs app by default will not resize when in a WebXR session (including an emulated one). This does make sense, but it's not a standard behavior shared among different engines/frameworks, so as we work towards a more elegant solution, just don't resize your window after entering VR for now.

felixtrz commented 1 month ago

In case you are interested, I am currently working on a migration of IWE on top of the newly released IWER: https://github.com/meta-quest/immersive-web-emulation-runtime to retire the outdated and heavily-modded webxr-polyfill currently powering IWE. I will keep this issue in mind as I build out the MR portion of the new IWE

marwie commented 1 month ago

I understand the confusion, but this feature is supposed to simulate how passthrough works in headset - a video layer showing your physical surroundings that will always be in the background of the scene you are rendering. Similarly for IWE, the injected layer renders the emulated physical environment (planes, meshes, etc.) and they will always be under your scene rendering.

I understand your reasoning and it makes sense - I'd just suggest to have (1) an option to disable it so it's less confusing when debugging and (2) maybe instead of solid bright colors use a pattern texture or different patterns that can act like background noise in the real world.

I'm pretty sure I didnt resize yes :)

Maybe it's already planned or you work on it but it would also be cool to have some predefined models or maybe whole room scans (loaded from real data) that can be used for testing

I havent had a chance to look into the new emulator yet besides seeing the announcement but will do so :) Thank you for your work!