Mapping Controller buttons in VR mode to trigger events

[rghv96]

Hi @andreasplesch There is a feature suggested by @npolys that when we are in VR mode we should be able to use controller buttons to perform additional functions apart from zooming in like different navigation modes, and custom functionality on the webpage by sending events.

I wanted a design review by you on how to implement this feature. I initially thought of adding code in VRControllerManager.

if ( controllers.left.gamepad.buttons[ 1 ].pressed )
        {
            const eventData = {
                controllerType : "left",
                buttonIndex    : 1,
                buttonState    : "pressed"
            };
            const customEvent = new CustomEvent( "myCustomEvent", { detail: eventData } );
            x3dom.debug.logInfo( "customEvent 1 triggered" );
            document.dispatchEvent( customEvent );
        }

This thing works. I can maybe improve the code design to avoid multiple if checks. However, I wanted to double-check with you

• what eventData to send?
• is x3dom library right place to implement custom functionality? I think we can call https://developer.mozilla.org/en-US/docs/Web/API/Navigator/getGamepads from the HTML page itself.

[andreasplesch]

https://developer.mozilla.org/en-US/docs/Web/API/Gamepad has an overview. It turns out that there are no events for button presses for gamepads. There is a proposal for such: https://chromestatus.com/feature/5989275208253440 which points to a polyfill for such events which would probably worth checking out. Currently, one is supposed to poll, eg. ask for button presses every frame. So VRControllerManager update() or so seems appropriate. It really depends on how the buttons are used to determine the best payload for a custom event. Probably it is ok to just include all available data (controller data, headset pose, button, target? ..) which may be relevant. The situation is quite different from how touch and mouse devices are supported. Overall, I think, I would recommend to go with the polyfill (https://github.com/MozillaReality/gamepad-plus/blob/master/README.md) if possible as it may become a standard solution and web devs are probably more comfortable with events than with RAF or gameloops. Then see how far one can get with runtime or dom methods and determine what additional runtime methods may be useful. A typical use for button would be to activate a wand for selection and pointing. So this might be a good concrete goal to explore custom use of controllers.

[andreasplesch]

Reading, it looks like the polyfill is not really maintained although it may still be a good option, if it works with modern devices.

[rghv96]

From what I understand we would have to use polyfill event listener at these two places

• Webpage: to perform a custom action when a button on the gamepad is pressed.
• x3dom library: to perform actions related to x3dom like "activate a wand for selection and pointing"

Am I right? In terms of implementation, can you suggest how to start the code of integrating gamepad-plus in x3dom? Do we need to override the gamepad API? I could just find this demo https://github.com/MozillaReality/gamepad-plus/blob/master/demo.js which doesn't help much with how to begin.

Also, other issues which I think would be fixed:

• controllers are not visible in VR mode when using the actual headset. They are visible when using the browser extension.
• When shifted to VR mode when using the actual headset, the x3d canvas on HTML freezes into a constant left and right view. It should be updated like the headset display(either stereo view or single view)

[andreasplesch]

I think the idea would be to only use the webpage javascript. And from there use runtime methods and dom manipulation to affect the x3d scene.

I was hoping that the poly-fill would extend the existing the gamepad API, so no other changes would be needed. If you load the gamepad poly-fill before x3dom, does the gamepad still work in VR ?

I seem to remember that there are two inlines added for the controller rendering ? The extension does its own rendering.

The x3d canvas is used to generate the left and right views for the headset, so there is no way around that. One could hide it and render a second x3d canvas with a mono view but that probably impacts performance.

[rghv96]

I seem to remember that there are two inlines added for the controller rendering ? The extension does its own rendering.

I hardcoded this this.leftInline.setAttribute( "url", "https://x3dom.org/download/assets/vr/vive.glb" ); but it did not work. I could not see the controller.

The x3d canvas is used to generate the left and right views for the headset, so there is no way around that. One could hide it and render a second x3d canvas with a mono view but that probably impacts performance.

but the left and right views are frozen, we need them to be updated on the webpage also.

[andreasplesch]

I seem to remember that there are two inlines added for the controller rendering ? The extension does its own rendering.

I hardcoded this this.leftInline.setAttribute( "url", "https://x3dom.org/download/assets/vr/vive.glb" ); but it did not work. I could not see the controller.

The Inlines should already be there, at the end of the scene. You can check with devtools. Did the controllers show up before the button movement changes ? If so, now the button presses change the viewarea._movement which is used to place the controller inlines with the matrix trafo. Perhaps this is out of sync now and controllers are always behind the viewpoint. Perhaps try switching in update():

    this._updateControllerModels( viewarea, vrFrameData.controllers );
    this._updateMatrices( viewarea, vrFrameData.controllers );

You could change all the render false calls to render true to make sure the display never gets switched off.

The x3d canvas is used to generate the left and right views for the headset, so there is no way around that. One could hide it and render a second x3d canvas with a mono view but that probably impacts performance.

but the left and right views are frozen, we need them to be updated on the webpage also.

That may be a browser preference ? Does the web page display on the monitor freeze with three.js/AFrame XR as well ? Is there a separate canvas which is assigned to the XR device ? https://immersive-web.github.io/webxr/explainer.html#inline-sessions seems relevant

[rghv96]

Controller Visibility issue

• Why is render false for controllers? this.leftInline.setAttribute( "render", "false" ); in VRControllerManager Does this line explicitly marks the controller invisible?

  The Inlines should already be there, at the end of the scene. You can check with devtools.

• How to check inline element of x3d scene using devtools?

Did the controllers show up before the button movement changes ?

I dont think so. I will try the switching of function calls and render = true change today when I access the headset.

Inline session issue

@npolys shared this https://github.com/immersive-web/webxr/issues/1317 which might be related to this issue?

[andreasplesch]

---on the phone---

On Thu, Apr 27, 2023, 2:22 AM Raghav Sethi @.***> wrote:

Controller Visibility issue

• Why is render false for controllers? this.leftInline.setAttribute( "render", "false" ); in VRControllerManager Does this line explicitly marks the controller invisible?

The render flag controls rendering. It is false by default because the Inlines are always attached even for non XR rendering. It is set to true later in the code.

The Inlines should already be there, at the end of the scene. You can check with devtools.

• How to check inline element of x3d scene using devtools?

devtools - elements - x3d - scene - matrixtransform

Did the controllers show up before the button movement changes ?

I dont think so. I will try the switching of function calls and render = true change today when I access the headset. Inline session issue

@npolys https://github.com/npolys shared this immersive-web/webxr#1317 https://github.com/immersive-web/webxr/issues/1317 which might be related to this issue?

Yes, this looks like it is the same issue.

—

Reply to this email directly, view it on GitHub https://github.com/x3dom/x3dom/issues/1267#issuecomment-1524826752, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABPCT246RXFFAQHOTD3AHIDXDIGA7ANCNFSM6AAAAAAW26P6CQ . You are receiving this because you were mentioned.Message ID: @.***>

[rghv96]

@andreasplesch regarding the controller issue

x3dom.VRControllerManager.prototype._addControllerModels = function ( controllers )
{
    if ( this.modelsAdded )
    {
        return;
    }

    if ( controllers.left )
    {
        const url = this._getControllerModelURL( controllers.left.type, "left" );
        this.leftInline.setAttribute( "url", url );
    }

    if ( controllers.right )
    {
        const url = this._getControllerModelURL( controllers.right.type, "right" );
        this.rightInline.setAttribute( "url", url );
    }

    this.modelsAdded = true;
};

It seems when _addControllerModels is called for first time controllers.left and controllers.right are false which don't update the URL in the HTML element and then in the further calls we don't update URL as modelsAdded becomes true. However, the controller works as the zoom is working.

I hardcoded

if ( this.modelsAdded )
    {
        return;
    }

    this.leftInline.setAttribute( "url", "https://x3dom.org/download/assets/vr/vive.glb" );
    this.rightInline.setAttribute( "url", "https://x3dom.org/download/assets/vr/vive.glb" );

which works fine then

Also, can we have a default controller when the model is not present in VRControllerManager._controllers? I was thinking to use HTC vive as it has same left and right models. Or create a new default controller model and add it in VRControllerManager._controllers.

So doubts are why is controllers.left false and how to proceed with default controller. We can also change the design of _addControllerModels when to mark modelsAdded = true.

[andreasplesch]

It sounds like your controller is not any of the types in ._controllers ? If so, check what vrFrameData.controllers.leight.type reports and add the model to ._controllers. Yes, there could be a default controller url: https://www.turbosquid.com/Search/3D-Models/free/controller . For now, just this._controllers.htc-vive.left or so. Perhaps it will start to make sense to make this customizable, eg. make it possible to easily add controllers to the list of _controllers. It is not great to have x3dom.org urls hardcoded in there. I am thinking x3dom could come with a very small .glb (as dataurl) as default controller. Perhaps just a few polygons. https://poly.pizza/m/U72ixRcVIK seems good but still 170kb https://poly.pizza/m/6365MG_Pr_f is only 30kb but a gamepad https://poly.pizza/m/z76Vm9mH2f is 70kb. draco compressed 12kb, attached, 18kb or so as dataurl. VR Controller.zip

[rghv96]

@andreasplesch

I know about the hardcoded controllers. Ideally there should be a config which can be updated independent of code. x3dom can make call to the config to fetch them. But since x3dom itself is not a deployed service and instead a library I am not sure how to separate the URLs from code. For now we can return the HTC vive as default model since its already published on x3dom

Can you also answer my other doubt

It seems when _addControllerModels is called for first time controllers.left and controllers.right are false which don't update the URL in the HTML element and then in the further calls we don't update URL as modelsAdded becomes true. However, the controller works as the zoom is working.

[andreasplesch]

It seems when _addControllerModels is called for first time controllers.left and controllers.right are false which don't update the URL in the HTML element and then in the further calls we don't update URL as modelsAdded becomes true. However, the controller works as the zoom is working.

This sounds like your controller is not any of the types in ._controllers ? What does vrFrameData.controllers.right.type report ?

[rghv96]

@andreasplesch I don't think that's the problem because I have added this hardcode of htc-vive in this function.

x3dom.VRControllerManager.prototype._getControllerModelURL = function ( type, side )
{
    if ( this._controllers[ type ] === undefined )
    {
        return this._controllers[ "htc-vive" ][ side ];
    }

    return this._controllers[ type ][ side ];
};

I will double check value of vrFrameData.controllers.right.type when I access a headset. I am unable to see debug logs in VR mode so it's taking time to debug.

[andreasplesch]

And with this change the controllers still do not display ?

With regards to user provided custom models for controllers, I think it will be best to only provide a runtime function: setXRControllerURL( url, side ) which will set an x3dom namespace global x3dom.defaultController[side][side]["URL"] . There are a number of such defaults in x3dom.

[andreasplesch]

For a simple, low-poly controller model, we should take advantage of the higher level primitives in x3d, rather than use gltf. Here is a 590 byte x3d which would suffice to show controller orientation:

https://github.com/andreasplesch/Library/blob/gh-pages/web3d/x3dom/XRController.x3d

Github pages can directly serve it:

https://andreasplesch.github.io/Library/web3d/x3dom/XRController.x3d

https://andreasplesch.github.io/Library/Viewer/index.html?url=https://andreasplesch.github.io/Library/web3d/x3dom/XRController.x3d

https://create3000.github.io/x_ite/playground/?url=https://andreasplesch.github.io/Library/web3d/x3dom/XRController.x3d

Here is the dataurl:

data:text/plain;base64,PFgzRD4KICA8U2NlbmU+CiAgICA8U2hhcGU+CiAgICAgIDxBcHBlYXJhbmNlIERFRj0nYTAnPgogICAgICAgIDxNYXRlcmlhbCBkaWZmdXNlQ29sb3I9JzAuMSAwLjEgMC4xJyBzcGVjdWxhckNvbG9yPScwLjA1IDAuMDUgMC4wNScvPgogICAgICA8L0FwcGVhcmFuY2U+CiAgICAgIDxDeWxpbmRlciBoZWlnaHQ9JzE2JyByYWRpdXM9JzIuNScgc3ViZGl2aXNpb249JzgnLz4KICAgIDwvU2hhcGU+CiAgICA8VHJhbnNmb3JtIHRyYW5zbGF0aW9uPScwIDggLTInPgogICAgICA8U2hhcGU+CiAgICAgICAgPEFwcGVhcmFuY2UgVVNFPSdhMCcvPgogICAgICAgIDxFeHRydXNpb24gCiAgICAgICAgICAgICAgICAgICBjcm9zc1NlY3Rpb249Jy0wLjI1IC0yLCAtMC41IDAsIC0wLjI1IDIsIDAuMjUgMiwgMC41IDAsIDAuMjUgLTIsIC0wLjI1IC0yJwogICAgICAgICAgICAgICAgICAgc3BpbmU9JzAgMCAwLCAtMy41IDAgMS41LCAtNSAwIDUsIC0zLjUgMCA4LjUsIDAgMCAxMCwgMy41IDAgOC41LCA1IDAgNSwgMy41IDAgMS41LCAwIDAgMCc+CiAgICAgICAgPC9FeHRydXNpb24+CiAgICAgIDwvU2hhcGU+CiAgICA8L1RyYW5zZm9ybT4KICA8L1NjZW5lPgo8L1gzRD4=

[rghv96]

@andreasplesch

I found out using chrome debugger that xrFrame.session.inputSources in X3DCanvas returns an empty array because of which ultimately controllers are not getting passed to VRControllerManager. If I wait for a second and then visit the breakpoint(xrFrame.session.inputSources) again in the next frame the XRInputSource array is present for the left and right controllers. This is for the current setup I have at the lab. I will also try to use a different setup tomorrow. Can you also check if you can see the controllers on this page https://metagrid2.sv.vt.edu/~raghavsethi/basic.html if you have a VR setup?

Regarding the default controller, I again replaced the URL in the inline element using Chrome dev tools and found that I am not able to load your x3d controller. On replacing the URL with the Vive headset URL https://x3dom.org/download/assets/vr/vive.glb I was able to see the Vive controller on the webpage. I have tried replacing the URL with

• https://andreasplesch.github.io/Library/web3d/x3dom/XRController.x3d

• https://andreasplesch.github.io/Library/Viewer/index.html?url=https://andreasplesch.github.io/Library/web3d/x3dom/XRController.x3d

• and the base64 dataURL but could not see the x3d controller model

[andreasplesch]

Unfortunately, I do not have a vr setup currently.

If you use the dataurl you also need to set the "contentType" attribute to "model/x3d+xml" because the url does not the have the extension.

The scale and orientation of the model does not correspond to the vive.glb model. The vive.glb uses very small numbers.

I updated to x3d model to correspond to the vive model here:

https://andreasplesch.github.io/Library/Viewer/index.html?url=data:text/plain;base64,PFgzRD4KICA8U2NlbmU+CiAgICA8VHJhbnNmb3JtIHNjYWxlPScwLjAwMDMgMC4wMDAzIDAuMDAwMycgcm90YXRpb249JzEgMCAwIC0xLjU3JyB0cmFuc2xhdGlvbj0nMCAwIDAuMDAyJz4KICAgIDxTaGFwZT4KICAgICAgPEFwcGVhcmFuY2UgREVGPSdhMCc+CiAgICAgICAgPE1hdGVyaWFsIGRpZmZ1c2VDb2xvcj0nMC4wMSAwLjAxIDAuMDEnIHNwZWN1bGFyQ29sb3I9JzAuMDUgMC4wNSAwLjA1Jy8+CiAgICAgIDwvQXBwZWFyYW5jZT4KICAgICAgPEN5bGluZGVyIGhlaWdodD0nMTYnIHJhZGl1cz0nMi4nIHN1YmRpdmlzaW9uPSc4Jy8+CiAgICA8L1NoYXBlPgogICAgPFRyYW5zZm9ybSB0cmFuc2xhdGlvbj0nMCA4IC02LjQnIHNjYWxlPScwLjggMC44IDAuOCc+CiAgICAgIDxTaGFwZT4KICAgICAgICA8QXBwZWFyYW5jZSBVU0U9J2EwJy8+CiAgICAgICAgPEV4dHJ1c2lvbiAKICAgICAgICAgY3Jvc3NTZWN0aW9uPSctMC4yNSAtMiwgLTAuNSAwLCAtMC4yNSAyLCAwLjI1IDIsIDAuNSAwLCAwLjI1IC0yLCAtMC4yNSAtMicKICAgICAgICAgc3BpbmU9JzAgMCAwLCAtMy41IDAgMS41LCAtNSAwIDUsIC0zLjUgMCA4LjUsIDAgMCAxMCwgMy41IDAgOC41LCA1IDAgNSwgMy41IDAgMS41LCAwIDAgMCc+CiAgICAgICAgPC9FeHRydXNpb24+CiAgICAgIDwvU2hhcGU+CiAgICA8L1RyYW5zZm9ybT4KICAgIDwvVHJhbnNmb3JtPgogICAgPFRyYW5zZm9ybSB0cmFuc2xhdGlvbj0nMC4wMDMgMCAwJz4KICAgIAk8SW5saW5lIHVybD0nImh0dHBzOi8veDNkb20ub3JnL2Rvd25sb2FkL2Fzc2V0cy92ci92aXZlLmdsYiInIC8+CiAgICA8L1RyYW5zZm9ybT4KICA8L1NjZW5lPgo8L1gzRD4=

https://andreasplesch.github.io/Library/web3d/x3dom/XRControllerViveScaled.x3d

has the model.

as data url: data:text/plan;base64,PFgzRD4KICA8U2NlbmU+CiAgICA8VHJhbnNmb3JtIHNjYWxlPScwLjAwMDMgMC4wMDAzIDAuMDAwMycgcm90YXRpb249JzEgMCAwIC0xLjU3JyB0cmFuc2xhdGlvbj0nMCAwIDAuMDAyJz4KICAgICAgPFNoYXBlPgogICAgICAgIDxBcHBlYXJhbmNlIERFRj0nYTAnPgogICAgICAgICAgPE1hdGVyaWFsIGRpZmZ1c2VDb2xvcj0nMC4wMSAwLjAxIDAuMDEnIHNwZWN1bGFyQ29sb3I9JzAuMDUgMC4wNSAwLjA1Jy8+CiAgICAgICAgPC9BcHBlYXJhbmNlPgogICAgICAgIDxDeWxpbmRlciBoZWlnaHQ9JzE2JyByYWRpdXM9JzIuJyBzdWJkaXZpc2lvbj0nOCcvPgogICAgICA8L1NoYXBlPgogICAgICA8VHJhbnNmb3JtIHRyYW5zbGF0aW9uPScwIDggLTYuNCcgc2NhbGU9JzAuOCAwLjggMC44Jz4KICAgICAgICA8U2hhcGU+CiAgICAgICAgICA8QXBwZWFyYW5jZSBVU0U9J2EwJy8+CiAgICAgICAgICA8RXh0cnVzaW9uIAogICAgICAgICAgICAgICAgICAgICBjcm9zc1NlY3Rpb249Jy0wLjI1IC0yLCAtMC41IDAsIC0wLjI1IDIsIDAuMjUgMiwgMC41IDAsIDAuMjUgLTIsIC0wLjI1IC0yJwogICAgICAgICAgICAgICAgICAgICBzcGluZT0nMCAwIDAsIC0zLjUgMCAxLjUsIC01IDAgNSwgLTMuNSAwIDguNSwgMCAwIDEwLCAzLjUgMCA4LjUsIDUgMCA1LCAzLjUgMCAxLjUsIDAgMCAwJz4KICAgICAgICAgIDwvRXh0cnVzaW9uPgogICAgICAgIDwvU2hhcGU+CiAgICAgIDwvVHJhbnNmb3JtPgogICAgPC9UcmFuc2Zvcm0+CiAgPC9TY2VuZT4KPC9YM0Q+

[rghv96]

@andreasplesch I added your model using dev tools. It seems that the ring part of the controller is missing when used in the x3d scene. You can also try dev tools to verify the behaviour.

[andreasplesch]

x3dom-full is needed for the ring part.

[rghv96]

@andreasplesch

• I have the custom webxr x3dom script attached to the webpage so I believe the model would render correctly then once merged to master.

• I have modified the _addControllerModels function so that it can render left and right controllers independently. I believe it would solve the problem that if any controller is not visible in the first frame, it can be added later. I have tested it works on an Occulus headset connected to a PC. However, I tried this change on a standalone android-based headset, the controllers were not visible. Also the photosphere page also doesn't show the image on the standalone android-based headset. So I believe if the below code change is added to the x3dom branch which has fixed the rendering issue fixed, the controllers would be visible. Let me know if I need to create PR.

  x3dom.VRControllerManager.prototype._addControllerModels = function ( controllers )
  {
  if ( controllers.left && !this.leftModelAdded )
  {
      const url = this._getControllerModelURL( controllers.left.type, "left" );
      this.leftInline.setAttribute( "url", url );
      this.leftModelAdded = true;
  }
  
  if ( controllers.right && !this.rightModelAdded )
  {
      const url = this._getControllerModelURL( controllers.right.type, "right" );
      this.rightInline.setAttribute( "url", url );
      this.rightModelAdded = true;
  }
  };

[andreasplesch]

Thanks for all the testing.

Would the additional this.left/rightModelsAdded fields make this.modelsAdded unnecessary ? Eg. what happens if there is only one controller ? It seems redundant and a bit confusing to have both.

Also, please initialize these additional fields after https://github.com/x3dom/x3dom/blob/e61f881945e95e57af2590b64f33504eea01ad26/src/util/VRControllerManager.js#L8\

I did not quite follow the Android headset issues.

It is easy to build a custom x3dom-full.js, for testing or special purpose applications. Just run npm run build after cloning to your PC, or follow https://github.com/x3dom/x3dom/wiki/Online-Build-Instructions