Hand tracking?

[thisismygatekeeper]

Just want to discuss with you to see if this pixi live2d library supports hand tracking if (1) the uploaded live2d model has hand and/or arm model and (2) real time hand tracking is available (through leap motion js library, or mediapipe). Thanks!

[guansss]

Sorry for the late reply! And yeah, it's possible as long as the Live2D model provides precise parameters to control the hands, e.g. ARM_LEFT_ANGLE, THUMB_LEFT_ANGLE, so that you can manipulate them with hand tracking results.

The implementation may be pretty tough though, I've never seen any Live2D model that works with hand tracking. 😕

[thisismygatekeeper]

Thanks for the reply! This example looks very appealing https://www.youtube.com/watch?v=txqbmjUwGtc This looks to be direct control of the arm / hand angles rather than triggering some static expressions. So in principle we could control it using parameters like ARM_LEFT_ANGLE, THUMB_LEFT_ANGLE with coreModel.setParameterValueById in pixi live2d?

[guansss]

Yeah it's really cool, the motions are more complicated than I thought, looks like the model has defined additional poses like flipped hands and bent fingers, and turn them on and off according to hand tracking data.

Theorically, you can achieve the same result using this plugin, like I said, as long as there are parameters, you can control them. But this is more of a matter of Live2D itself than this plugin, as it's just a bridge between Pixi and Live2D, which does not affect the availablility of Live2D low-level APIs.