Open connorgmeehan opened 1 year ago
I have had a crack implementing this. I opted for the 2nd approach outlined in my initial feature request. Here's a live example [permalink]. TinyTokyo is included as a complex scene, just to show that the GL State is not messed up.
Here's the changes, any advice or feedback on the PR would be very appreciated. Bare in mind this is based on r151 but once I have this working and can pass it off to my team I can upstream my changes to the latest three.js revision.
I am having a few issues: (never mind solved with some later commits)
One final thing, not sure on the best way of disposing the textures or if anything will need to be done to cleanup the textures seeing as they're managed by webxr.
Would you be okay to continue in #26452 and mark this issue as a duplicate? In this way, we have the discussion about the WebXR Raw Camera Access Module at one place.
@Mugen87 that issue is more about the VideoTexture not working in that specific case, it's better to have the discussion about the Raw Camera Access Module here since @connorgmeehan has proposed some API changes.
@connorgmeehan good job on the implementation. I believe Option 1 is more coherent to the three.js api style, the WebXRManager.getCameraTexture()
function can just reuse an internal ExternalOpaqueTexture
object so it's not fully exposed to the user.
I agree with @marcofugaro on keeping it seperate, it looks like we have 2 issues:
camera-access
feature enabled breaks THREE.js' internal state. When rendering a scene the texture will be bound on the first rendered element (other issue).RawTexture
class to handle external textures and then integrate it with the WebXRManager's and the camera-access
texture. (this issue)I just realised that my solution wasn't a complete solution to camera feed access and can have rendering artifacts depending on what's rendered in the scene (due to the other issue) so I'm going to park this issue, try to investigate and fix the other issue and then come back to this once it's done.
Also, yep for when I get back I am happy to re-implement with your changes @Mugen87 and using the option 1 approach.
Bit of a delay as I got side tracked at work. Found some previous art that we can maybe replicate. I should (hopefully) be working on this next week and PRing of the latest. https://github.com/BabylonJS/Babylon.js/pull/14527
Description
I would like to be able to get a
Texture
class that references the camera feed from an XR session.Solution
User API
Proposal would be to be able to request the camera texture from the WebXRManager class, however this glosses over a lot of implementation detail and may not map to the WebXR access patterns very well.
Proposed Internal changes
I am currently having a stab implementing this, I have little experience with the internals of three.js so everything is open to change.
Would require the creation of a new THREE.js Texture class that receives the underlying WebGLTexture instead of initialising it. Lets call it
ExternalTexture
. In anExternalTexture
THREE.js is not responsible for creating / deleting the GPU data, only binding it and tracking its state. As this could introduce a lot of footguns it may be best to keep this class for internal use for now.Additionally, the texture provided by the WebXR api has some limitations. It is what the WebXR specs call an opaque texture. In summary it is:
requestAnimationFrame()
callback of theXRSession
deleteTexture
on this texture will result in anINVALID_OPERATION
error.requestAnimationFrame()
INVALID_OPERATION
error.This will probably require a second class that extends from
ExternalTexture
, lets call itExternalOpaqueTexture
. To navigate the requirement that the texture is only valid from within an XRSession'srequestAnimationFrame
callback, from what I see there are two options.Option 1: Manually passing in the new opaque texture each frame.
Option 2: Passing in a getter for the opaque texture in the constructor.
User API Usage
There are two ways the api might have to be used based on the limitations of an opaque texture (see above). These access patterns map to option1 and option 2 of how the underlying ExternalOpaqueTextureClass might work.
getCameraTexture
withinrequestAnimationFrame()
of the XR session or it will error. This object is valid only for therequestAnimationFrame
that it was called within and will error/warn if used otherwise.textures.setTexture2D
. If this is called from outside of an XR sessionsrequestAnimationFrame
it will be black.Edited for formatting and a bit of extra information
Alternatives
I have attempted to get the camera access texture using raw webgl binds and use three.resetState to clean the three state but it doesn't work.
Additional context
Draft proposal: WebXR Raw Camera Access Module Raw WebGL Example Raw WebGL Example, source code
26452
Our sales person sold a project that depends on this feature so I am going to try implement it to this spec (on r151 however). I would appreciate any advice, I know almost nothing about how three.js tracks the state internally so any help or advice would be much appreciated. If i can get it working for our project I can PR on the latest threejs version in my free time.