VTS FULLBODY TRACKING

🔗 website: vts-fullbody-tracking.gitlab.io

This plugin integrates full body tracking functionality using Mediapipe. It allows users to use tracked body parameters as inputs to control Live2D model in VTube Studio.

THIS PLUGIN IS UNDER DEVELOPMENT

This plugin may contain bugs and lack certain features.
Visit the Wiki or Website for more information on how the plugin works.
Join Discord Server to share tests and feedbacks, ask questions or get help.

How you can Help

• Live2D Rigger: Help understand how to effectively rig models for the body parts feature
• Live2D Vtuber: Experiment with usability and performance for movement and configuration in VTube Studio
• Developper: Contributions are welcome to improve this plugin

Troubleshooting

• Currently, there might be a latency of a few seconds
• Tracking input may be inaccurate and exhibit occasional jumps

Run the Plugin

Requirements: Window, VTube Studio, Camera

1. Download the executable from the releases page
2. Connect a Camera and Open VTube Studio.
3. Double-click on the executable file VTS_Fullbody_Tracking.exe to launch the plugin. A settings window will appear.
4. Select your camera and click on the 'Start Tracking' button
5. Allow the plugin in vtube Studio. Window displaying a preview of pose tracking will appear.
6. Configure your model's parameter settings, using plugin parameters as inputs. you can now choose body parts X, Y, Z coordinates, and visibility as inputs

📖 Wiki Section - Run the plugin

Settings

Peview Camera

Displays the image captured by the camera

default	Preview Camera

📖Wiki Section - Settings Window

Custom Parameters

This plugin add new controls for various body parts in Vtube Studio. \ Each body part has parameters for controlling its position and visibility.

• Arms: Shoulders, Elbow, Wrists
• Torso: Clavicles, Hips
• Legs: Knees, Ankles
• Feet: Heels, Feet Index
• Hands: Pinkies Knuckles, Index Knuckles, Thumb Knuckles

📖Wiki Section - List of Parameters

DEVELOPMENT

Instructions for developers who want to run the plugin from the source code.

Requirements

• Python 3.11

Install dependencies

pip install -r requirements.txt

note: The plugin use the method vts.vts_request.requestSetMultiParameterValue from the library pyvts. This method is not included in the latest released version 0.3.2. You will need it to run the plugin, which can be obtained from the current repository. Required method: https://github.com/Genteki/pyvts/blob/main/pyvts/vts_request.py l.246

Run Plugin

• Open Vtube Studio
• Start the plugin

python app.py

Build executable

 pyinstaller ./app.py -n VTS_Fullbody_Tracking-0.1.7 --add-data='models/*:models' --add-data='icon.png:.' -F -w 

Documentation

Mediapipe

mediapipe documentation -Available Pose Landmarker models
mediapipe documentation - landmarker python
Blog Research Google - On-device, Real-time Body Pose Tracking with MediaPipe BlazePose

VTube Studio

VTube Studio API: https://github.com/DenchiSoft/VTubeStudio

Documentation Python Library

mediapipe: https://pypi.org/project/mediapipe/
pyvts: https://genteki.github.io/pyvts/
pyinstaller: https://pyinstaller.org/en/stable/