Should we have a generic profile for hands?

[Manishearth]

AR devices like the Hololens may expose hands as input sources instead of physical devices.

Hands don't have buttons, though they'll still expose a select (and potentially a squeeze https://github.com/immersive-web/webxr/issues/876) event. This could be mapped to generic-trigger-squeeze, but it might be best to mark them separately, especially if we ever come up with a future inputSource.hand API that exposes articulated hand tracking.

It would be good to have a generic ready for this (generic-hand?), even if right now it just means "this device has a select/squeeze event".

See also: https://github.com/immersive-web/webxr-gamepads-module/issues/23: If we decide that hand inputs may/should not have a Gamepad object, it may be more important to differentiate them from gamepads.

cc @thetuvix @NellWaliczek

[NellWaliczek]

One other thing to consider in this: what would an ideal fallback list look like for this sort of situation? ['generic-trigger-squeeze,generic-hand`]? The reverse? Just one or the other? And how would a developer be expected to use that information?

[yl-msft]

There are proposals on OpenXR working group for hand tracking extensions, and translate the direction to here, the expected fallback list would be this: [ 'generic-hand-mesh', 'generic-hand-joints', 'generic-trigger-squeeze', 'generic-button' ], and each one contains super-set of hand tracking capabilities to the next.

[Manishearth]

/agenda to see what people think about a generic hands profile

[albertotono]

Hand tracking would be great, also in having exposed the generic-hand-joints' it would be helpful: https://vimeo.com/user58338277. So I would be finally able to easily place the ring of my wife's finger.

[thetuvix]

I have a conflict for today's CG call, but I am in favor of having a generic-hand profile that guarantees a common base for developers:

• “left” vs. “right” handedness
• A gripSpace (near palm position + orientation)
• A targetRaySpace (far targeting ray)
• onselectstart/onselect/onselectend events (for far interaction, e.g. air-tap)
• onsqueezestart/onsqueeze/onsqueezeend events (for near interaction, i.e. grasp)

The key gotcha in my mind remains developers who ignore onselect and/or onsqueeze and directly target the source's gamepad to fetch button 0 and 1 instead, which will limit app compatibility. To get ahead of that, we can be proactive in having our own samples always use the events rather than polling those gamepad buttons, and also ensuring that higher-level XR web engines and input libraries unify hand and controller input consistently at that layer.

[Artyom17]

FYI: for Quest we can have only "one button controller" (with the "pinch" gesture). Theoretically, we can have more (if we consider "index pinching", "middle pinching", "ring pinching", "pinky pinching" as separate buttons), but we don't have 'squeeze' gesture at the moment.

[Manishearth]

I feel like the best path forward here is a "generic-hand" profile, which has the following characteristics:

• Must have a select, like all input sources
• May have a squeeze
• May have a grip space
• Does not have a gamepad object (this can be changed later)

I'd like this to be very lightweight, with the main goal being that it's a signal to the author that we don't want to render a "normal" controller, rendering either just a hand or an abstract control point. I think if we do this we'd need to mention it in the core spec somewhere and also update this library to be okay with that string.

[Manishearth]

@toji thoughts? do you know how to make the changes to this library so it will accept the new generic?

[Artyom17]

My only concern is that the experience must be aware of that profile (if it won't be backed by the generic-...-controller profile). What is better - to have hands rendered as VR controllers and don't have anything?

[Manishearth]

I think it's better to do nothing by default, and experience can use something abstract if they wish. Basically, similar to what experiences should be doing for AR already.

For example, the WebXR paint example on Threejs can render controllers, but it also renders a blob of paint at the center of the controller. In AR (and ideally, for hand APIs), it only renders a blob of paint, which is sufficient to make things make sense.

[thetuvix]

Do we expect hands without a grip space? That would be a device that knows when you do a select press but does not otherwise know where the hand is. Is that strong enough to count as "generic-hand"?

@Manishearth: So far for controllers, the generic input profiles are more specific about letting apps know which inputs are actually present, rather than leaving various inputs to be optional.

In analogy then to "generic-trigger" and "generic-trigger-squeeze", the base profile with just select and grip space can be "generic-hand" and then squeeze-capable platforms can support "generic-hand-squeeze". That way, apps know based on the profile whether to map drag-and-drop actions to "onsqueeze" or "onselect".

[Manishearth]

generic-hand and generic-hand-squeeze seem fine to me.

[Artyom17]

What is the status of this issue? Is there an agreement? Seems like this would be a useful addition to the Hands Input proposal.

[Manishearth]

I feel like there's general agreement. A wrinkle is that https://github.com/immersive-web/webxr/issues/997 means that for many devices select and squeeze can overlap, and we may also need a generic-hand-squeezebutton or something where the squeeze is exposed as a button instead.

I would actually not add this to the hands input proposal, because you can have hand input devices that do not support articulated hand tracking, indeed this is the current state of devices that wish to support hand input in webxr :smile: