Body Data Abstraction

[james-h-stephens]

Creation of a device-agnostic body model for gesture recognition.

[joeabsoftdev]

Should we facilitate device agnosticism within the Body namespace or upstream to Body?

[james-h-stephens]

We will be creating a device-agnostic body for now. I think that we should be focusing on tracking the joints. Thoughts?

[GMelencio]

Yes - definitely the joints. It's precisely what we need for BVH, plus it seems to be what is most common to be tracked between devices, and if not we can interpolate the joint positions.

[GMelencio]

@james-h-stephens Upon having abstracted joints, we would need a device agnostic way of determining the position of body parts esp. extremities. To that effect, do you agree that we'd need to find a formula to address Issue # 1?

[james-h-stephens]

@GMelencio So joints have Vector3 positions that represent them in 3-space. I am unsure I understand what more in the way of positional data you desire.

[GMelencio]

it's not about the positional data per-se, it's about joint positions relative to each other

Think about it this way, something as simple as fingers - how can we tell if the angle of flexion at a joint is increasing or decreasing if, throughout the period of measurement, the hand with said finger is rotating slightly each time joint positions are reported?

[james-h-stephens]

Angular measurement is independent of positional data. For example, the two angles below are not occupying the same positions, yet we can compare the angles between them.

Much the same way, rotation (changing of positional data around some axis) will not affect our ability to compare two angle. If we have points: A (0, 1, 0), B (0, 0, 0), and C (1, 0, 0)

we know that angle ABC = 90 degrees

If we then have points: D (0, 1, 0), E (0, 0, 0), and F (0, 0, 1)

we know that angle DEF = 90 degrees

DEF is simply the rotation of ABC about the Y axis at 90 degrees.

[GMelencio]

@james-h-stephens Thank you for that explanation. That solves it for many joints, how will this work for the joints that have more than one degree of freedom (i.e. hip, neck, thumb MP, spine, etc).

I see now that my question did not reveal the true complexity of the problem. By way of example this may clarify: using only the joint positions - the shoulder joint being the primary joint of flexion, how can we tell if a user is doing an Uppercut punch (or jab works too) versus a Right Hook punch? (Why? Well, aren't these "gestures"?)

My guess is we'd have to look at the difference in angle and/or distance between the humerus vs joints in the body.

Here's our ("bvh body demo")[http://abantech-demo.cloudapp.net/samples/human/bodyanimation.html] for your reference.

[joeabsoftdev]

My sense as result of your comments is that conversion would be best positioned as a pair of layers between the middleware and UI/UX I/O systems, on both sides of the middleware. That way, additions and updates to Efficio, Efficio::Body, the Engine, and so forth, will be standardized across devices. Specializing to devices would take place in converter layers, which I believe would be not only logical - especially where our 3-space control scope gains complexity, but also elevate performance. Would we be on the right track with the following?:

[joeabsoftdev]

Upstream Converter:

//Outline for Greg and/or James

//include statements

//1. Recognize device //2. Recognize high-level device settings substantially affecting middleware structure //3. Select path to convert UX/UI script to targeted, genericized Efficio middleware ecosystem // a. Write optimized pathway for each device // b. Write optimized pathway for high-effect high-level device specifications // c. Convert narrow system to inclusive system

---

Downstream Converter:

//Outline for Greg and/or James

//include statements

//1. Recognize device //2. Recognize high-level device settings substantially affecting middleware structure //3. Select path to convert Efficio middleware ecosystem into individual device/program UI/UX // a. Write optimized pathway into each device // b. Write optimized pathway into high-effect high-level device specifications // c. Convert general system into individual system

[joeabsoftdev]

Finally, if so, should we get started with the C++ of it in a week or two?

[joeabsoftdev]

Last, to James' point on tracking joints, the multimaps approach does this, but I want to deal with syntax more before I push it up to you. Part of that depends on converters, or something similar, if that's the right way. The main approach has shared pointers to arrange this - the way those are used can change depending on our math approach to joint swivels.

[joeabsoftdev]

James' most recent Master commit generally implements the upstream Leap of my approach impression above. Abstracting-joint-data is currently almost entirely device-agnostic. Should we merge to test abstraction for Leap?

[james-h-stephens]

@joeabsoftdev Which branch is it? I do not see it in or current branches

[joeabsoftdev]

As mentioned previously, body .cpp seems missing. I'm pretty sure I scripted it and pushed it to Github with errors so you could read it. The file seemed missing after that. I'll search again and rewrite it if necessary

[joeabsoftdev]

I searched for body.cpp using the taskbar document search Best match for documents and Documents - This PC Search in File Explorer. Both returned a .html file with body twice in the name. Body.cpp was not returned.

[joeabsoftdev]

I searched for body.cpp in GitHub Abantech/Efficio. The file was not returned in the results list.