Bone-based tracker structure

[zachkinstner]

The plugin currently sends a pose (position + rotation) for elbow, wrist, palm, and all finger bones. See Tracker.cpp and the device descriptor for full details.

Let's use this ticket to discuss the naming/organization of these points. Here's the current structure, as a starting point for everyone to review.

---

Raw paths are in the format:

/com_osvr_LeapMotion/Controller/tracker/NUMBER

...where NUMBER is an integer from 0 to 45.

---

Semantic paths map raw paths to human-readable names, and they use this hierarchy:

/com_osvr_LeapMotion/Controller/semantic/arms
  /left, /right
    /elbow *
    /wrist *
    /hand
      /palm *
      /thumb, /index, /middle, /ring, /pinky
        /metacarpal *
        /proximal *
        /intermediate *
        /distal *

...where the comma-separated items represent separate branches in the tree. Items marked with * contain the pose data.

2 arms with (3 points for elbow, wrist, palm + 5x4 bones) each = 2x23 = 46 points.

---

Alias paths can re-map the semantic paths however we want, creating a path that starts at the root. For now, the structure copies the semantic tree, placed at the root like /arms/.../....

[zachkinstner]

In my simple Unity test client, using OSVR's Unity ClientKit, I'm finding that it takes a long time (~50 seconds) to obtain interface connections for all of these tracker paths. The Unity app is not responsive during this time.

Once I have the connections, it takes another minute or so for all these connections to start providing data. The client's cubes (which each follow one pose) start reacting, one by one, to the hand movements during this time. Eventually, they are all reacting as expected, with the 46 cubes showing the elbow/hand/bone structure.

[zachkinstner]

Tracker::getOsvrQuaternion now converts the bone basis matrices to quaternions using Eigen. Note that the Leap::Matrix class uses a row-major format, while Eigen uses (by default) a column-major format.

My tests in Unity show that the right hand's rotations look correct, but the left hand's do not. The positions for both hands look correct. @jdonald, can you think of why this might occur? For example, are the hand/bone basis matrices mirrored/flipped somehow for left vs. right hands?

[jdonald]

For example, are the hand/bone basis matrices mirrored/flipped somehow for left vs. right hands?

Well, there is this note in the API regarding Hand::basis() :

      * Note: Since the left hand is a mirror of the right hand, the
      * basis matrix will be left-handed for left hands.

Could that explain it?

[zachkinstner]

Thanks @jdonald! I should have checked the docs before asking.

The commit referenced above fixes the issue, using notes from the Bone::basis() docs:

The bases provided for the right hand use the right-hand rule; those for the left hand use the left-hand rule. Thus, the positive direction of the x-basis is to the right for the right hand and to the left for the left hand. You can change from right-hand to left-hand rule by multiplying the z basis vector by -1.

[zachkinstner]

I'm finding that it takes a long time (~50 seconds) to obtain interface connections

I moved this comment to the issue referenced above.

---

Let's use this ticket to discuss the naming/organization of these points. [...] Alias paths can re-map the semantic paths however we want

I sidetracked this ticket a bit, but I think we can get back to this topic now.

[GabrielHare]

First question: is OSVR assuming that there will be at most 1 left hand and 1 right hand in the field of view? If so, how are frames that are not consistent with this assumption handled?

[zachkinstner]

@GabrielHare Currently, the plugin loops through the hand list, assigning each hand to either the left or right tracker outputs. If there are multiple left hands, for example, the last one in the list would be sent.

The plugin does assume that there will be at most one left hand and one right hand. For VR usage, 0-2 hands seems like the vast majority of usage scenarios. Technically speaking, the plugin must set a fixed number of tracker channels at compile-time. To support 3+ hands, we could add more tracker channels (and the associated semantic+alias API paths), and then only use them if there are additional hands.

Since the "standard" API paths for getting hands are /me/hands/left and /me/hands/right (set in the device descriptor), the app/user would most likely not be looking for additional hands.

[GabrielHare]

After a long delay, which I really regret... I have more feedback.

My concern here is that choosing the last hand from all left hands or the last hand from all right hands will effectively reduce the reliability of the Leap service since spurious background hands can replace the user's actual hands.

On the application side this is usually addressed by hiding spurious hands using a visual fade based on confidence, since the spurious hands usually are not near interactive elements. However, I do not think that a cut on hand confidence will be safe, since in some poses the user's hand can have a low confidence.

After talking this through with Alex and Wilbur the best approach seems to be to allow for later refactors. To this end:

• Please create a module that is responsible for deciding which left & right hand to choose. This way, the decision process can be changed, or reduced to Leap API calls such as "GetBestLeftHand()".

That being said, the best approach seems to be based on the longevity of tracking of real hands. The logic for the left hand would be as follows:

• Suppose that in the previous frame, a hand has been identified as the user's left hand, and is present in the current frame. Then all other left hands in the frame will be marked as non-candidates.
• Suppose that in the previous frame, no hand was identified as the user's left hand. In the current frame, no hands that have been marked as non-candidates could be identified as the user's left hand. However, any candidate (new) left hand that appears could be the user's left hand. If there are multiple candidate left hands, then choose the one that is closest.
• This logic could be problematic if there is an object yielding spurious left hand that persists for multiple frames, thereby preempting the user's hands.

[jdonald]

This does seem like quite a bit of work to get around limitations of a spec that was sketched out but not really vetted months ago. Even if we resolve this with a best-hand-chooser, there are long term costs to maintaining something that has different high-level behavior from our existing API and plugins. Could we rename the semantic paths to make this a non-issue?

Is it possible to have lists in the path? e.g. /com_osvr_LeapMotion/Controller/semantic/arms[1]/hand/...

[zachkinstner]

Plugins define a fixed number of output channels (see my comment above), so we would need to plan ahead for which scenarios we want to handle. For each hand, we have 23 "tracker" channels, plus several "analog" channels to pass hand-related values (like the confidence level). Even if we supported two left hands and two right hands, we may still need some logic to choose the best hands in a (very unusual) 3+ left/right hand scenario.

With the current delay to "connect" to these channels in Unity (see issue #6), these channels could create a significant amount of additional wait time. I'm not sure what other penalties there may be for adding extra channels.

[jdonald]

Oh I see. The number of output channels is a limitation of the plugin interface, and would be there regardless of how we map our path tree aliases. Oh well, that brings us back to Gabe's suggestions for introducing GetBestLeftHand()-related logic.

[zachkinstner]

First pass at the hand-selection logic is available in the HandSelector class in the commit above. Includes a //TODO note:

Improve this selection process, probably by looking at the hands over time. For example, if a candidate hand maintains a higher confidence than the current winner, across several frames of data, then that candidate could become the new winner.

I need to shift the usage of this new class into the core part of the plugin (instead of just the "tracker" module), because other areas (notably, the "analog" module) also need to know about the active hand(s).

[zachkinstner]

I'm not entirely sure how to test the hand selections:

• Using some debugging output (currently commented-out in the code), I can see when certain hand-selection events occur, but the scenarios that lead to these events are hard to reproduce reliably.
• My computer speakers usually get picked up as hands, so that is sometimes helpful for generating low-confidence hands ;)
• Using my simple Unity testing app, I can see the hands' tracker positions/orientations. So far, they are appearing as expected.

[zachkinstner]

The new LeapData class wraps the Leap Controller and two (left and right) HandSelector instances. It is created within ControllerDevice, and provides a consistent way for the various plugin modules (tracker, analog, etc.) to obtain this information.

[GabrielHare]

Hand selection logic based on confidence looks good to me.