search

FomTarro / vts-heartrate

Heartrate Plugin for VTube Studio
MIT License
7 stars 1 forks source link

readme

We finally have a logo!

VTS-Heartrate

A VTube Studio plugin that allows for connectivity between heart rate monitors (HRM) and VTube Studio!

Pulsoid Verified Badge

Download the latest version here! Or, in the release section of this repo.

Working Example

Features

💓 Support for many heart rate monitors with pulsoid.net, hyperate.io, ANT+ and even Fitbit!

💓 Configurable model tinting that scales with pulse!

💓 Automatic expression and hotkey triggering at desired heartrate thresholds!

💓 Custom tracking parameters for pulse and breath!

💓 Support for Live2D Items!

💓 NEW! Plugin API so that you can build your own apps that consume or write heartrate data!

Table of Contents

About

This plugin is developed by Tom "Skeletom" Farro. If you need to contact him, the best way to do so is via Twitter.

If you're more of an email-oriented person, you can contact his support email: tom@skeletom.net.

Usage

Getting up and running is relatively straightforward. The plugin will automatically connect to VTube Studio on launch. From there, do the following steps:

A comprehensive video tutorial has also been provided as reference.

Heartrate Input Methods

In the interest of being widely accessible, this plugin features a wide set of possible input methods, which you may freely switch between at any time.

Test Slider

The slider is primarily useful for quick testing of different heartrate values. The slider range is from 0 to 255.

Read from File

This input method allows you to read heartrate data from an external file. The file must simply contain the numeric heartrate value in plain text. File path must be absolute. Useful if you have another program that can output heartrate data.

API WebSocket

This input method allows you to read heartrate data from the Input API. For more information about the API, consult the API Documentation.

Pulsoid

Pulsoid Verified Badge

Pulsoid is a free third-party app for Android/iOS which allows for easy, reliable connectivity to a wide set of heartrate monitors via the Bluetooth of your mobile device.

Once you have a Pulsoid account, you can use this input method to collect heartrate data from the service.

By clicking the 'Login' button in the plugin, you will be asked to grant this plugin permission to connect to your account. You will then be given an 'Authentication Token' which you must paste into the plugin.

ANT+

ANT+ is a low-power protocol supported by many sport and fitness sensors.

This input method allows for direct connection to your ANT+ device, provided that you have a USB receiver plugged in.

By clicking the 'Refresh' button, this plugin will begin a continuous scan for devices that output heartrate data. Then, simply select one from the dropdown and click the 'Connect' button.

Please note that this plugin is not an officially licensed or certified affiliate of the ANT+ Brand.

HypeRate

HypeRate is a free third-party app for Android/iOS which allows for easy, reliable connectivity to a wide set of heartrate monitors via the Bluetooth of your mobile device.

Once you have a HypeRate account, you can use this input method to collect heartrate data from the service.

Fitbit

Fitbit is wearable fitness device brand, with an app for Android/iOS.

In order to use your Fitbit with vts-heartrate, you will need to download a special companion app onto the Fitbit device itself.

To do this, select your watch model from the dropdown. Then scan the 'Download the Companion App' QR code with your phone, which will take you to the Fitbit App Gallery listing for the companion app version that is compatible with your model. Follow the prompts on your phone to install this companion app via the Fitbit phone app.

On the phone app, beneath the 'Install' button, there will be sections labelled as 'Permissions' and 'Settings'. Make sure to grant all requested permissions. Then navigate to 'Settings', where you will be prompted for Your Local IP, which is provided to you by the vts-heartrate PC app. Upon entering that information, your Fitbit watch should finally begin sending heartrate data to vts-heartrate after a short delay.

If you ever need to return to the 'Settings' menu again, in order to update Your Local IP, simply scan the 'Download the Companion App' QR code again.

This setup also requires that your phone is on the same network as the PC running vts-heartrate, because your phone is actually the device relaying information between the Fitbit and the PC.

A list of compatible Fitbit models can be found in the FAQ at the bottom of this README.

Outputs

Art Mesh Tinting

This output will gradually tint the matched Art Meshes the desired color, based on your current heartrate. For example, you can use this to make your model's face flushed for workouts, or run cold during horror games.

Art Meshes are matched as long as their names or tags contain any of the provided list of text to match. For example, providing the text 'head,mouth,neck' will match Art Meshes with names like 'forehead', 'outermouth2', and 'leftneckside1'. Text to match must be comma separated and should not contain spaces.

If you are unsure of what your Art Meshes are named, a great web-tool was developed by Hawkbar called VTubeStudioTagger, which offers an intuitive way to discover the names of your model's Art Meshes.

Additionally, as of version 1.3.0, you can select specific Art Meshes by clicking the 'Select Art Meshes in VTube Studio' button, which will open a prompt in VTube Studio.

Working Example

Automatic Expression Triggering

This output will cause an Expression to activate or deactivate when the current heartrate is above or below a given threshold, based on the selected behavior settings.

For example, the configuration in the provided image will cause the angry Expression to automatically activate when the heartrate exceeds 120 BPM, and will deactivate when the heartrate falls back beneath 120 BPM.

Working Example

Automatic Hotkey Triggering

This output will cause a Hotkey to be triggered when the current heartrate is above or below a given threshold, based on the selected behavior settings.

For example, the configuration in the provided image will cause the panic Item Scene to automatically toggle when the heartrate exceeds 120 BPM, and will toggle again when the heartrate falls back beneath 120 BPM.

Working Example

Custom Tracking Parameters

This plugin outputs fifteen custom tracking parameters for use. They are as follows:

The following parameters are primarily suited for controlling Live2D props, but you may also find them useful of your model:

For more information on how to integrate these tracking parameters into your model, please refer to the Official VTube Studio documentation.

Custom Parameter setup

Live2D Items

There are several Live2D Item assets available which are already configured to take advantage of the above custom parameters. You can find them at the 7M|C Shop. These assets packs are:

These packs include things such as sweat drips, EKG monitors, steamy breath puffs and more.

Of course, if you're up the the task, you can also make your own Live2D Items using these custom parameters, as well. You can find the complete guide to making Live2D Items in the VTube Studio Manual.

Profiles

As of version 1.2.0, vts-heartrate supports profiles.

Profiles allow you to have multiple output configurations for the same model. For example, you may want one profile where you go blue in the face for Horror Games, and another profile where you flush red and activate a sweating expression for Workouts.

You can even copy profile configurations from one profile to another, even across models. This is useful if you have multiple variations of the same base model, such as different outfits.

When a model is loaded in VTube Studio, a DEFAULT profile is automatically loaded along with it. User-created profiles must be loaded manually.

API

As of version 1.2.0, vts-heartrate features its own Plugin API, so that you can build your own apps that consume or write heartrate data! That's right, this VTube Studio plugin now supports plugins of its own.

There are three underlying API endpoints, all accessible via WebSocket: the Data API, the Events API, and the Input API.

Data API

The Data API is a read-only endpoint accessible at ws://localhost:<your chosen port>/data. Upon connecting to this endpoint, your WebSocket will receive a message containing current heartrate and output parameter data once per frame (60 times per second).

The message structure is as follows:

{
    apiVersion: "1.0",
    messageType: "DataResponse",
    timestamp: 1656382245785
    data: {
        heartrate: 103,
        parameters: {
            vts_heartrate_bpm: 103,
            vts_heartrate_bpm_hundreds: 1,
            vts_heartrate_bpm_ones: 3,
            vts_heartrate_bpm_tens: 0,
            vts_heartrate_breath: 0.29349666833877563,
            vts_heartrate_linear: 1,
            vts_heartrate_pulse: 0.994240403175354,
            vts_heartrate_repeat_1: 0.27418050169944763,
            vts_heartrate_repeat_5: 0.07212083041667938,
            vts_heartrate_repeat_10: 0.3203545808792114,
            vts_heartrate_repeat_20: 0.12136854976415634,
            vts_heartrate_repeat_30: 0.4221900999546051,
            vts_heartrate_repeat_60: 0.714303731918335,
            vts_heartrate_repeat_120: 0.932239294052124,
            vts_heartrate_repeat_breath: 0.29349666833877563
        },
        tints: [
            {
                baseColor: { r: 255, g: 128, b: 128, a: 255 },
                currentColor: { r: 255, g: 200, b: 200, a: 255 },
                matchers: ['head', 'face']
            },
            {
                baseColor: { r: 255, g: 128, b: 128, a: 255 },
                currentColor: { r: 255, g: 200, b: 200, a: 255 },
                matchers: ['mouth', 'neck']
            }
        ]
    }
}

For more information about the output parameters, consult the Custom Tracking Parameter Documentation and Art Mesh Tinting Documentation.

Event API

The Event API is a read-only endpoint accessible at ws://localhost:<your chosen port>/events. Upon connecting to this endpoint, your WebSocket will receive a message containing event information every time an Expression Trigger or Hotkey Trigger is triggered.

For Expressions, the message structure is as follows:

{
    apiVersion: "1.0",
    messageType: "ExpressionEventResponse",
    timestamp: 1656382245785
    data: {
        threshold: 120,
        heartrate: 121,
        expression: "angry.exp3.json",
        behavior: 2,
        activated: true
    }
}

The complete list of possible behavior values is as follows:

UNKNOWN = -1,    
ACTIVATE_ABOVE_DEACTIVATE_BELOW = 0,
DEACTIVATE_ABOVE_ACTIVATE_BELOW = 1,
ACTIVATE_ABOVE = 2,
DEACTIVATE_ABOVE = 3,
ACTIVATE_BELOW = 4,
DEACTIVATE_BELOW = 5,

For Hotkeys, the message structure is as follows:

{
    apiVersion: "1.0",
    messageType: "HotkeyEventResponse",
    timestamp: 1656382245785
    data: {
        threshold: 120,
        heartrate: 121,
        hotkey: "Blush",
        behavior: 1
    }
}

The complete list of possible behavior values is as follows:

UNKNOWN = -1,    
ACTIVATE_ABOVE_ACTIVATE_BELOW = 0,
ACTIVATE_ABOVE = 1,
ACTIVATE_BELOW = 2,

Input API

The Input API is a read and write endpoint accessible at ws://localhost:<your chosen port>/input. Upon connecting to this endpoint, your WebSocket will be able to write heartrate data for use with the API WebSocket input method.

Authenticating Your Plugin

First, you will need to authenticate your client before you are granted write permission. In order to authenticate, you must first request a token by sending a message with the following structure:

{
    messageType: "AuthenticationRequest",
    data: {
        pluginName: "My Heartrate Plugin",
        pluginAuthor: "Skeletom",
        pluginAbout: "A plugin for connecting a third-party service."
    }
}

The user will then be prompted to either Approve or Deny access to this plugin. If the request is approved, the API server will then respond with a message containing a token.

You only need to do this step if your plugin has not already been granted a token. If it has a token already, you can skip to the next step.

The message structure is as follows:

{
    apiVersion: "1.0",
    messageType: "AuthenticationResponse",
    timestamp: 1656382245785
    data: {
        pluginName: "My Heartrate Plugin",
        pluginAuthor: "Skeletom",
        token: "e404d80b-c160-4af4-a0b9-9c0159f3010e",
        authenticated: false
    }
}

It is strongly recommended that you save this token so that your plugin may use it again in the future. Finally, you must now submit an authentication request using the token, by sending a message with the following structure:

{
    messageType: "AuthenticationRequest",
    data: {
        pluginName: "My Heartrate Plugin",
        pluginAuthor: "Skeletom",
        token: "e404d80b-c160-4af4-a0b9-9c0159f3010e"
    }
}

Assuming you have provided a token that the user has granted access for, the API Server will finally respond with a message indicating that your plugin is fully authenticated, and that you may begin writing heartwrate data.

The message structure is as follows:

{
    apiVersion: "1.0",
    messageType: "AuthenticationResponse",
    timestamp: 1656382245785
    data: {
        pluginName: "My Heartrate Plugin",
        pluginAuthor: "Skeletom",
        token: "e404d80b-c160-4af4-a0b9-9c0159f3010e",
        authenticated: true
    }
}

Writing Heartrate Data

Once your plugin is fully authenticated, you can write heartrate data sending a message with the following structure:

{
    messageType: "InputRequest",
    data: {
        heartrate: 78
    }
}

Upon successfully writing data, the API Server will echo your message back to you as an InputResponse, to confirm that it has been received.

Errors

In the event that something goes wrong, such as the API Server receiving a message it cannot parse, or a message from an unauthenticated client, your WebSocket will receive an error message.

The message structure is as follows:

{
    apiVersion: "1.0",
    messageType: "ErrorResponse",
    timestamp: 1656382245785
    data: {
        errorCode: 403,
        message: "This client is not authenticated!"
    }
}

The complete list of possible errorCode values is as follows (they are HTTP status codes):

OK = 200,
BAD_REQUEST = 400,
FORBIDDEN = 403,
SERVER_ERROR = 500,

FAQ

Q: "What devices are supported?"

A: This plugin supports any device that is supported by

Unfortunately, Fitbit has not provided a way to develop or install third-party apps on their latest models (Versa 4 and later), so the latest devices cannot be supported at this time.

Roadmap

Planned features include the following: