Unable to change viewpoints in VR mode(WebXR branch)

[rghv96]

I tried WebXR integration by testing the webxr GitHub branch on my webpage https://metagrid2.sv.vt.edu/~raghavsethi/ which has an x3dcanvas and buttons to change viewpoints. I am using an Oculus Go headset with a hand controller. Once, I entered the VR mode, I could not change viewpoints using buttons on the webpage. When the trigger button on the hand controller is pressed, the scene moves to random nearby viewpoints in VR Mode. I could not find the logic inside the code on how it was happening. I would like to move through different viewpoints inside the VR mode. If anyone can guide me on how to debug/provide direction to the solution, that would be great.

[andreasplesch]

A pretty long time ago I experimented with webVR navigation with viewpoints. Here is the history of commits in my webVR_viewpoint branch:

https://github.com/andreasplesch/x3dom/commits/webVR_viewpoint/src/util

Things have changed but perhaps this could be a starting point.

[rghv96]

Hi @andreasplesch

I read the file src/util/VRControllerManager.js. I am unable to find the logic from where the default viewpoint is getting picked-up at. I have read that x3d default viewpoint position is (0,0,10), which I could verify in the file src/nodes/Navigation/Viewpoint.js. However, the initial viewpoint is not positioned at (0,0,10) when we enter VR mode. I cannot find the exact coordinates of the initial viewpoint because the debug logs don't work properly in VR mode.

I am trying to place the initial viewpoint at the origin when we enter VR mode. I tried updating the default viewpoint position to (0,0,0). It works in the normal mode but not in the VR. In VR mode, the viewpoint is still not at the origin.

I tried to understand the code by reviewing a debug log in VR mode, which was generated from src/gfx_webgl.js, but I could not make any progress.

Can you please help me find the code where the initial viewpoint in VR mode is defined?

[andreasplesch]

Ok, let me see. I may try to add comments to VRControllerManager.js . I did not check how much VRControllerManager.js changed in the latest WebXR branch but perhaps not too much.

[andreasplesch]

https://github.com/x3dom/x3dom/blame/webxr/src/util/VRControllerManager.js#L139

.fit() is called when switching to XR, on first update(). fit() then just moves away from the current position by half the scene size in the viewing direction (of the headset). Not quite sure why.

continued ..

[andreasplesch]

While there are quite a few modifications to VRControllerManager.js for viewpoint stuff, these are only related to navigation with controllers and not headset related.

The viewpoint related headset view updates are in Viewarea.js: https://github.com/andreasplesch/x3dom/commit/31a4c8290f3935bb25c8beb0943aeaf81c69bad5

The .getViewMatrices function is called every frame. It turns out that if there is vr data viewpoints get ignored completely. The only change is to multiply in the Viewpoint viewmatrix to combine it with the headset viewmatrix. This apparently was enough.

[rghv96]

• I think the line https://github.com/x3dom/x3dom/blame/webxr/src/util/VRControllerManager.js#L139 has some connection with the logic that if you are in VR mode and press a button on the controller, it moves to a different point in the 3D scene. If I remove the line meaning the movement variable becomes zero, the headset initializes at the origin, but the controller behavior does not happen.
• I saw your commit https://github.com/andreasplesch/x3dom/commit/31a4c8290f3935bb25c8beb0943aeaf81c69bad5. You multiplied VR left and right view matrices with the viewpoint matrix. Is the intention of the commit to attach the headset to the current active viewpoint? I tried the code and observed some weird behavior. Also, can you recommend me any resource on how to understand when and where I need to multiply these matrices(trans, rotational)

[andreasplesch]

• I think the line https://github.com/x3dom/x3dom/blame/webxr/src/util/VRControllerManager.js#L139 has some connection with the logic that if you are in VR mode and press a button on the controller, it moves to a different point in the 3D scene. If I remove the line meaning the movement variable becomes zero, the headset initializes at the origin, but the controller behavior does not happen.

Yes, that sounds right. If you are planning to use navigation with controllers in addition to the headset, it will make sense to look more closely at the other changes as well.

• I saw your commit andreasplesch@31a4c82. You multiplied VR left and right view matrices with the viewpoint matrix. Is the intention of the commit to attach the headset to the current active viewpoint? I tried the code and observed some weird behavior. Also, can you recommend me any resource on how to understand when and where I need to multiply these matrices(trans, rotational)

Yes, I think that was the intention. The ._transMat and ._rotMat I think are used for mouse/keyboard navigation updates relative to the viewpoint. Perhaps that use also has changed somewhat. Also, it is possible that webXR now delivers a different kind of view matrix which needs a different treatment to combine with a viewpoint.

[andreasplesch]

It looks like the webxr branch is using the inverse.matrix for the headset transform as delivered by webxr:

https://developer.mozilla.org/en-US/docs/Web/API/XRView/transform

https://github.com/x3dom/x3dom/blob/webxr/src/nodes/Navigation/Viewpoint.js#L187 shows how the viewpoint matrix is derived, also as an inverse of the transform. So in principle this should be compatible.

To combine it is necessary to make sure both are using the same reference coordinate system/space. I think that means WebXR should use 'local'.

I used viewarea.getViewpointMatrix() which actually takes into account the global Transform of the viewpoint and therefore returns a global coordinate system. But I think that still means 'local' for the WebXR space.

[andreasplesch]

WebXR documentation seems to recommend to dynamically update the webxr reference space with a rigid transform to modify the 'viewpoint'. So this would be another option.

https://developer.mozilla.org/en-US/docs/Web/API/XRReferenceSpace#usage_notes https://developer.mozilla.org/en-US/docs/Web/API/WebXR_Device_API/Movement_and_motion#applying_the_user_inputs

https://developer.mozilla.org/en-US/docs/Web/API/XRRigidTransform/XRRigidTransform

The position of the rigid transform would be the viewpoint position, and the rotation the viewpoint orientation quaternion.

Conceptually, rather than modifying the X3D view to accommodate the headset, one would modify the headset view to take into account the X3D viewpoint.

A problem would be transitioning between viewpoints which is nicely animated, and could not be easily reproduced,

[rghv96]

Hi @andreasplesch

Thanks for sharing the links. I made the following change in X3DCanvas.js to dynamically update the viewpoints inside VR mode. I took the position of the existing viewpoint and provided its details in the offsetTransform. I checked the code at https://github.com/x3dom/x3dom/blob/webxr/src/X3DDocument.js#L529 on how to find the viewpoint. However, on testing the changes, when I entered the VR headset, the viewpoint seemed slightly off from the supposed current viewpoint. I could verify it from the logs, also. Can you please review the code and suggest any changes?

You can test it using the headset/ webxr extension on: https://metagrid2.sv.vt.edu/~raghavsethi/ https://metagrid2.sv.vt.edu/~raghavsethi/photoxr-local.html

navigator.xr.requestSession( "immersive-vr", { requiredFeatures: [ "local-floor" ] } ).then( ( session ) =>
    {
        session.requestReferenceSpace( "local" ).then( ( space ) =>
        {
            this.xrReferenceSpace = space;
            // var mat_view = this.doc._viewarea.getViewMatrix().inverse();
            var mat_view = this.doc._viewarea.getViewMatrix();

            var rotation = new x3dom.fields.Quaternion( 0, 0, 1, 0 );
            rotation.setValue( mat_view );
            var rot = rotation.toAxisAngle();
            var translation = mat_view.e3();

            const offsetTransform = new XRRigidTransform( {x: translation.x, y: translation.y, z: translation.z},
                {x: rot[ 0 ].x, y: rot[ 0 ].y, z: rot[ 0 ].z, w: 1} );
            this.xrReferenceSpace = this.xrReferenceSpace.getOffsetReferenceSpace( offsetTransform );

            x3dom.debug.logInfo( "XRReferenceSpace Offset position: " +  translation.x.toFixed( 5 ) + " "
                + translation.y.toFixed( 5 ) + " " + translation.z.toFixed( 5 ) + "\" " +
                "orientation=\"" + rot[ 0 ].x.toFixed( 5 ) + " " + rot[ 0 ].y.toFixed( 5 ) + " "
                + rot[ 0 ].z.toFixed( 5 ) + " " + rot[ 1 ].toFixed( 5 ) );
        })
    })

Also, to calculate mat_view, I removed the inverse() as it gave me opposite values for the z position coordinate. I am new to computer graphics and view matrix concepts, so not sure if it's correct or not.

A problem would be transitioning between viewpoints which is nicely animated, and could not be easily reproduced,

In normal mode, the animation happens smoothly between different viewpoints. So it should be possible in VR mode also somehow?

[andreasplesch]

One thing I noticed is that XRRigidTransform directly takes the x,y,z,w quaternion for orientation, not axis angle. The documentation also says the q. needs to be normalized.

So

rotation.normalize();
const offsetTransform = new XRRigidTransform( {x: translation.x, y: translation.y, z: translation.z},
                {x: rotation.x, y: rotation.y, z: rotation.z, w: rotation.w} );

should be more correct.

For me, the chrome extension still reported 1.60m for the headset position y. Not sure where this number is coming from.

[npolys]

likely checking the bound navigation info; 1.6 m is the default avatarSize for eye height above ground...

On Thu, Feb 16, 2023 at 2:09 PM Andreas Plesch @.***> wrote:

One thing I noticed is that XRRigidTransform directly takes the x,y,z,w quaternion for orientation, not axis angle. The documentation also says the q. needs to be normalized.

So

rotation.normalize();const offsetTransform = new XRRigidTransform( {x: translation.x, y: translation.y, z: translation.z}, {x: rotation.x, y: rotation.y, z: rotation.z, w: rotation.w} );

should be more correct.

For me, the chrome extension still reported 1.60m for the headset position y. Not sure where this number is coming from.

— Reply to this email directly, view it on GitHub https://github.com/x3dom/x3dom/issues/1242#issuecomment-1433584300, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2TSM5RFX4VBIHDUNSGLSLWXZ3PZANCNFSM6AAAAAAUJG3JHE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Nicholas F. Polys, Ph.D.

Director of Visual Computing Virginia Tech Research Computing

Affiliate Professor Virginia Tech Department of Computer Science https://people.cs.vt.edu/~npolys/ https://people.cs.vt.edu/~npolys/

[andreasplesch]

Makes sense, and seems correct for a viewpoint location of 0,0,0 .

It would be the x3dom computed view matrix which takes into account the avatar height, which then gets translated into the XRRigidTransform which yields a new xr reference space which is reported by the extension.

[rghv96]

Hi @andreasplesch

I have updated the code to use quaternion for orientation. You can verify it on my web pages shared in previous comments. Also, what are your thoughts on

A problem would be transitioning between viewpoints which is nicely animated, and could not be easily reproduced,

In normal mode, the animation happens smoothly between different viewpoints. So it should be possible in VR mode also somehow?

Also, @npolys suggested exploring the feature of using controller orientation for direction and controller buttons to navigate in VR mode. It shouldn't take headset orientation into consideration. I will explore the code to see how to work on this feature. I believe it also involves "animation" throughout the navigation movement.

[andreasplesch]

Viewpoint animation possibly would still be possible but will require deeper changes since now the viewpoint is not represented by the view matrix only by the offset.

But the first step would be to change viewpoints at all in xr. Just generating a new offset and new space should do it.

vrcontrollermanager would be the place to experiment with controller navigation.

[rghv96]

But the first step would be to change viewpoints at all in xr. Just generating a new offset and new space should do it.

Not sure what you mean, but we can achieve this, right? with the offset code change mentioned in this thread, we can change the viewpoint outside VR mode and then enter it.

[andreasplesch]

Yes, changing the viewpoint outside and then enter VR is already possible with the initial offset. Some VR experiences have a concept of gates you can enter to be transported to a new place or world as well.

[rghv96]

Hi @andreasplesch

I have added the support to move in VR mode using controllers. It's live on https://metagrid2.sv.vt.edu/~raghavsethi/. I followed the logic mentioned in the PR https://github.com/x3dom/x3dom/pull/948

The feature worked fine when I tested using the extension. @npolys When I used actual Oculus VR headset controllers, the experience was not perfect, probably because the sensors were not arranged correctly in the lab.

[andreasplesch]

Great!

[rghv96]

Hi @andreasplesch

What should be the next steps? Should I raise a PR to merge these changes to the webxr branch?

[andreasplesch]

It would be great to submit a PR against the webxr branch to allow contributors to be informed and decide if they are ok with these or suggest improvements. Additionally, PR will go through eslint coding style checks and allow anybody to adopt the code as desired.

A goal for x3dom adoption would be that XR support is sufficiently general to work on most devices, or gracefully fail. Perhaps this will actually require spec. guidance with regards to navigation through controllers, similar to the spec. guidance on the use of a pointer device/mouse or keyboard.

[rghv96]

@andreasplesch where can I add the documentation regarding the navigation through controllers?

[andreasplesch]

You can use the wiki in your x3dom fork, or in mine: https://github.com/andreasplesch/x3dom/wiki

[rghv96]

@andreasplesch please review my wiki changes https://github.com/rghv96/x3dom/wiki/WebXR-Changes. Please let me know if I should add more content.

[andreasplesch]

Thanks. Perhaps you could expand on how the features are implemented, conceptually, on a higher level ?

[rghv96]

I have updated the wiki and added implementation details.

[rghv96]

Hi @andreasplesch As you can see on the PR, the scaleFactor is set to 1 for adding movement when the button is pressed on controllers. It can create issues when the x3d scene is very small. Ideally, the movement should work just like zoom in the non-VR mode which I believe uses zNear and zFar to have zoom according to the scale of the scene.

I checked the implementation of x3dom.DefaultNavigation.prototype.zoom in DefaultNavigation.js and have improved the controller movement by

x3dom.VRControllerManager.prototype._getViewAreaZoom = function ( viewarea )
{
    var navi = viewarea._scene.getNavigationInfo();
    var viewpoint = viewarea._scene.getViewpoint();
    var d = ( viewarea._scene._lastMax.subtract( viewarea._scene._lastMin ) ).length();
    d = Math.min( d, viewpoint.getFar() );
    d = ( ( d < x3dom.fields.Eps ) ? 1 : d ) * navi._vf.speed;
    const zoomAmount = 1;

    return d * ( zoomAmount ) / viewarea._height;
};

const cDirection = this._getControllerDirection( pose );
const scaleFactor = this._getViewAreaZoom( viewarea );

viewarea._movement = viewarea._movement.add( cDirection.multiply( scaleFactor ) );

I have normalised cDirection also first. Then multiply it with viewArea zoom. However, I don't know how to set the zoomAmount as DefaultNavigation.zoom is ultimately getting called by Runtime.zoom and I dont know from where Runtime.zoom is getting called or how the zoomAmount is passed with what value.

Please suggest how to initialize zoomAmount. You can test the changes on https://metagrid1.sv.vt.edu/~raghavsethi/ . Please let me know if should make a new pull request to webxr branch again.

[andreasplesch]

https://github.com/x3dom/x3dom/blob/be808673a29d32bee9fa9f323ce60d3692c71af0/src/Viewarea.js#L403 is where DefaultNavigation.zoom (or TurntableNavigation.zoom) gets called. And viewarea zoom apparently only gets called from runtime.zoom. runtime.zoom is not used internally. It is a public method for custom functionality.

I think that means you can treat zoomAmount in VR as a fudge factor to achieve comfortable zooming with continued controller button pressing at default navi. speeds. The X3D spec. says 1 m/s at a speed of 1 as far as I remember.

[n-polys]

How does X3dom handle collision and terrain following? Ie intepreting mouse movements in walk mode... Ie How do we get to avatars world space from location from a 2d drag?

On Mon, Mar 13, 2023, 11:51 AM Andreas Plesch @.***> wrote:

https://github.com/x3dom/x3dom/blob/be808673a29d32bee9fa9f323ce60d3692c71af0/src/Viewarea.js#L403 is where DefaultNavigation.zoom (or TurntableNavigation.zoom) gets called. And viewarea zoom apparently only gets called from runtime.zoom. runtime.zoom is not used internally. It is a public method for custom functionality.

I think that means you can treat zoomAmount in VR as a fudge factor to achieve comfortable zooming with continued controller button pressing at default navi. speeds. The X3D spec. says 1 m/s at a speed of 1 as far as I remember.

— Reply to this email directly, view it on GitHub https://github.com/x3dom/x3dom/issues/1242#issuecomment-1466407512, or unsubscribe https://github.com/notifications/unsubscribe-auth/AYQT72LH6LF6C27KMVLLBQTW3466LANCNFSM6AAAAAAUJG3JHE . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[MacrayBlackhand]

There is a pretty good example of drag and drop movement example on the examples page.

That is where I started with my drag and drop functionality. I got that working and then set to modify it to do all the other stuff I wanted.

Once you take a look at that, let me know if you have any specific questions and I'll be glad to let you know what I did.

As for "trailing" and Avatar with a Viewpoint, that took a bit more doing.  I'd be happy to answer specific questions, when/if you have them.

Steve

---- On Wed, 15 Mar 2023 09:12:55 -0500 n-polys @.***> wrote ---

How does X3dom handle collision and terrain following? Ie intepreting mouse movements in walk mode... Ie How do we get to avatars world space from location from a 2d drag?

On Mon, Mar 13, 2023, 11:51 AM Andreas Plesch @.***> wrote:

https://github.com/x3dom/x3dom/blob/be808673a29d32bee9fa9f323ce60d3692c71af0/src/Viewarea.js#L403 is where DefaultNavigation.zoom (or TurntableNavigation.zoom) gets called. And viewarea zoom apparently only gets called from runtime.zoom. runtime.zoom is not used internally. It is a public method for custom functionality.

I think that means you can treat zoomAmount in VR as a fudge factor to achieve comfortable zooming with continued controller button pressing at default navi. speeds. The X3D spec. says 1 m/s at a speed of 1 as far as I remember.

— Reply to this email directly, view it on GitHub <https://github.com/x3dom/x3dom/issues/1242#issuecomment-1466407512&gt;, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYQT72LH6LF6C27KMVLLBQTW3466LANCNFSM6AAAAAAUJG3JHE&gt; . You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, https://github.com/x3dom/x3dom/issues/1242#issuecomment-1470084134, or https://github.com/notifications/unsubscribe-auth/AWTX3ZXE5YA3YQ5V37VYVMDW4HE6PANCNFSM6AAAAAAUJG3JHE. You are receiving this because you are subscribed to this thread.

[andreasplesch]

see also #1186

[rghv96]

Hello @andreasplesch

I have followed the approach mentioned in DefaultNavigation.moveforward and written the code below

if ( controllers.left.gamepad.buttons[ 0 ].pressed ) {
            const pose = controllers.left.pose;
            const cDirection = this._getControllerDirection(pose);
            const scaleFactor = this._getViewAreaZoom(viewarea);

            let avatarRadius = 0.25;
            const navi = viewarea._scene.getNavigationInfo();

            if ( navi._vf.avatarSize.length > 2 )
            {
                avatarRadius = navi._vf.avatarSize[ 0 ];
            }

            var speed = 5 * viewarea._deltaT * navi._vf.speed;
            var yRotRad = ( viewarea._yaw / 180 * Math.PI );
            var xRotRad = ( viewarea._pitch / 180 * Math.PI );

            // var currViewMat = view.getViewMatrix();
            // view._flyMat = view.inverse();
            //
            // const fMat = view._flyMat.inverse();
            // const fMat = viewarea._flyMat.inverse();
            const fMat = viewarea.getViewMatrix();

            // check front for collisions
            viewarea._scene._nameSpace.doc.ctx.pickValue( viewarea, viewarea._width / 2, viewarea._height / 2, viewarea._lastButton );

            if ( viewarea._pickingInfo.pickObj )
            {
                const dist = viewarea._pickingInfo.pickPos.subtract( fMat.e3() ).length();
                x3dom.debug.logInfo( "reached pickObj, dist = " + dist.toFixed( 4 ) );

                if ( dist <= 2 * avatarRadius )
                {
                    x3dom.debug.logInfo( "Collision at dist =" + dist.toFixed( 4 ) );
                }
                else
                {
                    viewarea._movement = viewarea._movement.add( cDirection.multiply( scaleFactor ) );
                    viewarea._eyePos.x -= Math.sin( yRotRad ) * speed;
                    viewarea._eyePos.z += Math.cos( yRotRad ) * speed;
                    viewarea._eyePos.y += Math.sin( xRotRad ) * speed;
                }
            }
            else
            {
                viewarea._movement = viewarea._movement.add( cDirection.multiply( scaleFactor ) );
            }
        }

I am confused about fMat. For this line fMat = viewarea._flyMat.inverse(); , I am getting error because of null value of viewarea._flyMat.

I then randomly assigned fMat = viewarea.getViewMatrix() , I can go inside the if loop if ( viewarea._pickingInfo.pickObj ) however I am getting a large value of distance and thus not able to detect collision.

If you can help me with fMat, I can progress further. Basically, I need to understand how to initialize viewarea._flyMat or fMat

[andreasplesch]

Let's see. .moveForward takes view, and view is a viewarea. It is called from viewarea.moveFwd() which can call different implementations of .moveForward depending on navigation mode. .moveFwd() is called after arrow up key press. I think this is only for game mode.

In other modes (walk, fly) the front collision check occurs in .navigateTo() here:

https://github.com/x3dom/x3dom/blob/83ce36310eda365a35ece8391e9a8c3f78800c38/src/nodes/Navigation/modes/DefaultNavigation.js#L460

This pattern may be more appropriate since it does not use _flymat . .navigateTo actually computes _flymat during navigation. Does the VR controller navigation use .navigateTo() ?

Perhaps this helps.

[rghv96]

@andreasplesch

VR controller navigation is pretty basic and doesn't use .navigateTo()

The approach x3dom/src/nodes/Navigation/modes/DefaultNavigation.js is more or less the same.

dist = viewarea._pickingInfo.pickPos.subtract( ??? ).length(); I tested below things to check what to subtract from pickPos viewarea.getViewMatrix().inverse().e3(); viewarea.getViewMatrix().e3();

However, I noticed the issue is that my viewMatrix is not getting updated in VR mode and is fixed to the initial viewpoint. In the normal mode, in-game navigation the view matrix gets updated because of which value of dist gets updated.

Any idea how to ensure viewarea.getViewMatrix() gets updated? Ultimately I think we have to use viewarea.getViewMatrix() to calculate dist

[andreasplesch]

viewMatrix normally gets updated in .navigateTo here:

https://github.com/x3dom/x3dom/blob/83ce36310eda365a35ece8391e9a8c3f78800c38/src/nodes/Navigation/modes/DefaultNavigation.js#L519

or here in game mode:

https://github.com/x3dom/x3dom/blob/83ce36310eda365a35ece8391e9a8c3f78800c38/src/nodes/Navigation/modes/DefaultNavigation.js#L355

[andreasplesch]

And .navigateTo gets called every tick:

https://github.com/x3dom/x3dom/blob/be808673a29d32bee9fa9f323ce60d3692c71af0/src/Viewarea.js#L253

[rghv96]

@andreasplesch

• I found out by experimenting on x3d scenes that needNavAnim is mostly always false in .navigateTo DefaultNavigation when not in the game mode which means the view is not getting updated?
• Also, as .navigateTo gets called every tick, the view should have been updated in the VRControllerManager code also right? So ideally we don't need to update the view in VRControllerManager. If that's not the case, I thought of copying the logic of updating the view in VRControllerMigration from the .navigateTo in DefaultNavigation. However, that does not seem right to duplicate the logic and the logic is also complex to me. Do you think we could somehow call navigateTo inside VRControllerManager?

I put many hours to find a solution to the problem of providing collision support in VR and would appreciate it if you could provide a simple solution :)

[andreasplesch]

Thanks for trying to figure out the navigation logic.

.navigateTo gets called every tick but will not do anything I think if there is no mouse button pressed, eg. needNavAnim will be false.

It may work to emulate a pressed mouse button (._lastButton) and call .navigateTo from VRControllerManager.

For experimenting it is probably easier to duplicate code.

I think the path to a simple solution will be through a more complex one first ;)

[rghv96]

One doubt, we can get the current viewpoint in the VR mode by the changes in enterVR function in X3DCanvas file mentioned in https://github.com/x3dom/x3dom/issues/1242#issuecomment-1432146483. var mat_view = this.doc._viewarea.getViewMatrix(); So we should have got the updated viewarea in VRControllerManager?