hand-controls misaligned with HTC Vive controllers in 1.4.2

[wyzwon]

Description: As of commit 5b29ac2, the hands no longer line up with the HTC Vive controllers.

This is the log for the specific commit in which the issue first appears:

hand-controls: rotates models to match WebXR definition (fix #4848) (#5261)

It is a small commit that only changes handModelOrientation. It seems the Vive is not in agreement with the new WebXR standard. Is this an issue that exists at the level of SteamVR and their WebXR implementation? If so, maybe a workaround is possible in the mean time?

Note: Due to an issue with the build bot at the time, you won't see the issue in the cdn.jsdelivr.net link supplied for commit 5b29ac2 or any other commit before 7050d7b (the last April 19th commit). You'll have to remake the build yourself if you want to try those commits for any reason.

• A-Frame Version: 1.4.2 (5b29ac2 and up)
• Platform / Device: gen 1 HTC Vive / Windows 10 / Steam VR 1.26.3 (beta) / Chrome 114

[mrxz]

I was able to reproduce the issue with Valve Index controllers. The problem is entirely in A-Frame due to inconsistencies in how it handles various controls. When using hand-controls, there is a chain of dependencies as follows:

• hands-controls
• oculus-touch-controls, vive-controls, etc... (annoyingly enough it's missing valve-index-controls)
• tracked-controls
• tracked-controls-webxr (or in case of WebVR tracked-controls-webvr)

The hands-controls assumes that the object is in gripSpace in which case the rotation added in commit https://github.com/aframevr/aframe/commit/5b29ac2a24abec8209624c65e3fd335e9fcfecef is correct. However, the default space used by tracked-controls is targetRaySpace. Literally only oculus-touch-controls deviates from the default and configures tracked-controls to be gripSpace, all other controls components result in targetRaySpace, meaning the entire pose of the hand (position and rotation) is mostly bogus.

As a workaround you can manually override the space for tracked-controls:

<a-entity hand-controls="hand: left" tracked-controls="space: gripSpace"></a-entity>

[wyzwon]

Using tracked-controls="space: gripSpace seems to put their center of mass back in line with the physical controllers but the hands are still pitched 90 degrees forward relative to their old alignment.

[mrxz]

(...) the hands are still pitched 90 degrees forward relative to their old alignment.

Some screenshots might help, though it's probably hard to see how they should be orientated.

It's worth noting that the hand model is just straight and it seems neither orientations are perfect. At least for me on both Quest 2 and Valve Index the additional rotation does result in better orientation for the fingers. For example, if I stick out both my index fingers and try to make them touch, it lines up decently. Sticking out my thumbs also go in the right direction. But the wrists seem to "stick out" slightly upwards, as they don't angle down like they do in real life.

Without the additional rotation, the wrist seems to line up better, but the finger orientations no longer match. If I stick out my thumbs, the model does reflect this "gesture", but the thumbs direction are 90 degrees off from the actual direction my thumbs are pointing, same with the index fingers.

Does this align with what you're seeing with the Vive controllers?

[wyzwon]

1.4.1 and below orientation

1.4.2 orientation (with tracked-controls="space: gripSpace")

The new orientation beyond being an unexpected change, is way too clunky for a controller like the Vive controllers which are meant to be held more like a wand. The model's wrist sticks out of the base of my thumb in this rotation.

I was quite happy with how things were, it felt natural to grab things that way.

Fwiw, Valve takes a somewhat middle position based on their update to The Lab which gave the player robot hands. In their setup, the loop thing is about level with the index finger, and the handle passes through the edge of the wrist opposite the thumb.

[mrxz]

Thanks for the images. It's effectively the inverse problem a lot of older VR games have that were made with the VIVE wands when played with Index controllers. The tricky part is that WebXR doesn't provide a whole lot of information. The "rod" in grip space is given, but no clue as to how the user would hold this "rod" (as a wand, straight on or somewhere in between)

Wondering what the best approach for this is, but as a crude workaround you can manually set the rotation of the hand mesh after it has been loaded. Something like the following should do the trick:

const mesh = handControlsEl.getObject3D('mesh');
if(mesh) {
    mesh.rotation.x = 0;
} else {
    handControlsEl.addEventListener('object3dset', function waitForMesh(e) {
        if(e.detail.type !== 'mesh' || e.target !== handControlsEl) { return; }
        setTimeout(() => handControlsEl.getObject3D('mesh').rotation.x = 0, 0);
        handControlsEl.removeEventListener('object3dset', waitForMesh);
    });
}

Note the setTimeout is there since the hands-controls component first sets the mesh and then immediately updates the rotation, so without it the rotation would be overwritten by the component. (Alternatively you can make your own copy of hand-controls with the relevant changes made directly to it)

[dmarcos]

<a-entity hand-controls="hand: left" tracked-controls="space: gripSpace"></a-entity>

Should probably consolidate all in controllers in gripSpace?

[mrxz]

<a-entity hand-controls="hand: left" tracked-controls="space: gripSpace"></a-entity>

Should probably consolidate all in controllers in gripSpace?

From a consistency pov I agree, it's weird that only the oculus-touch-controls deviate from the rest. However, this might not be a simple change. It would impact all usages of xyz-controls, meaning controller models no longer align and any experience attaching raycasters under the assumption that they are in targetRaySpace would break.

Given that it's the hand-controls component that expects gripSpace to orientate the hand mesh, it might be an option to have it explicitly set this?

[wyzwon]

Do we have a way to distinguish the wands vs the index controllers? It would be ideal to detect and handle them differently or one will always be off (I assume anyways, I don’t have an index controller to try).

I could always add a manual switch in a settings tab if it came to it, but that’s a last resort.

[dmarcos]

FWIW we have separate components for vive-controls and valve-index-controls

[wyzwon]

That is nice and useful, it does solve the problem for something like an isolated museum exhibit where you know the hardware. I wouldn’t want to limit an online experience though.

[mrxz]

Under the hood the hand-controls uses these individual components, which you can check to see which one is actually active. For example:

if(handControlsEl.components('vive-controls').controllerPresent) {
   // Do something only for Vive controllers
}

[wyzwon]

Does “Vive-controls” mean just the wand? or does it trigger for index controllers as well? If that does distinguish them, shouldn’t that mean this is fixable under the hood?

I would certainly put in some sort of reactive correction if it worked and I need to for my project, but it would be nice to have it fixed everywhere, or are we considering the new orientation with wands to be the intended standard?

[mrxz]

Does “Vive-controls” mean just the wand? or does it trigger for index controllers as well?

It shouldn't trigger for the index controllers, but it does currently trigger for all HTC controllers (specifically the ones that prefix match htc-vive, like for example htc-vive-cosmos). A more rigorous check for the HTC Vive Wand would be something like:

handControlsEl.components['tracked-controls-webxr'].controller.profiles[0] === 'htc-vive'

(The Vive Wands should have the exact profile id "htc-vive")

If that does distinguish them, shouldn’t that mean this is fixable under the hood?

A "fix" can't really be applied to the vive-controls component directly. When using the vive-controls separate from hand-controls it should show you a 3D model of the controllers and be orientated in targetRaySpace. (I assume that in this configuration the controller models line up correctly?).

So only in the context of hand-controls does this offset make sense. Ideally we avoid the situation where the component needs different logic for each type of controller in addition to the information it gets from WebXR about the controller. I think the situation could perhaps be improved by using the gripSpace to know where the fingers/palm go and the (global) orientation to know where the wrist (likely) goes. Aligning the hand with those should make the behaviour more consistent across controllers (current and future). :thinking:

[wyzwon]

I definitely agree it wouldn't be ideal. It reminds me of the (thankfully mostly) gone days of browser identification and browser specific hack fixes, though if it ended up being the only way, I would consider it better than leaving the hands impractically bent for certain controllers. Hopefully your idea works out though.

I wonder if the group behind the standard is aware of this limitation?

When using the vive-controls separate from hand-controls it should show you a 3D model of the controllers and be orientated in targetRaySpace. (I assume that in this configuration the controller models line up correctly?).

Yes this does display correctly; in fact to get those pictures above I just stuck a pair of vive-controls components in the scene alongside the hand-controls.

[wyzwon]

So I opened up hand-controls and took a look around inside. I noticed that there is a "isVive" flag.

@mrxz I know you had this to say about the subject:

Ideally we avoid the situation where the component needs different logic for each type of controller in addition to the information it gets from WebXR about the controller.

but given that we are already doing the User-Agent string thing anyways, and a proper fix appears far off (out of our scope?), maybe we should consider just adding a re-adjustment for Vive wand controllers specifically right now to get things back to at least working order in the mean time.

[mrxz]

@wyzwon While I would still like to avoid additional logic for specific controllers, it might indeed be the easiest way forward for the time being. It would definitely help anyone designing experiences specifically for the Vive Wands using hand-controls.

On the flip-side it will make hand-controls behave less consistent. Say an experience tries to be controller agnostic and uses the thumb or pointing direction in some way (e.g. visualize some text above the thumb when the user thumbs-up). The hand model is aligned within the local space of the entity its on. If the Vive Wands would apply an (additional) rotation to the hand-model, it would no longer line up.

But in that regard the hand-controls already falls short (missing controllers, no generic-tracked-controller-controls for forward compatibility, assumes gripSpace while most controls components use targetRaySpace). So I'd say there's more value in providing a better OOB experience for the Vive Wands, than to aim for a mostly theoretical backwards compatibility with experiences relying on its inconsistent/broken behaviour.

@dmarcos What do you think? Essentially the hand-controls places the hand model in accordance with the WebXR grip space, but the way Vive Wands are held (as a wand) mean the hands visually don't match the physical hands orientation. Prior versions of A-Frame had a closer matching orientation. Idea is to reintroduce a similar rotation specifically for the vive wands.

[dmarcos]

hand-controls was supposed to be more like a demo on the early days inspired by Job Simulator not an accurate representation of user's hands. Not sure how relevant is today and how much usage hand-controls and Vive are getting these days. Haven't seen it much lately. In 2024 I definitively recommend using the actual controller model and hand tracking on supported headsets for an accurate hand representation. I've considered deprecating hand-controls to make clear what's the recommended path but not sure yet.

[wyzwon]

@dmarcos In that case, how do you recommend a developer today set up an experience designed for use in an unrestricted environment where any hardware could be used? Should we include every controller type and manually map events for each controller type?

I personally don't really care if the controller is a hand or a digital recreation of the controller that the user is using, but it is nice to have a consolidated component that bundles them all together for shared logic, maybe we need a new component that does that without the additional trouble of making them all share the same hand model and orientation?

[dmarcos]

@wyzwon we have laser-controls for that purpose. Single button (click) action. For something more advanced, you can follow laser-controls example. Add as many controllers as you want and map the buttons to any actions in your application.

There's now also hand-tracking-controls that it's becoming the common input method to all new headsets (e.g Vision Pro has no controllers)