Unreal Engine Project "Heartbeat" — Heart Rate Monitoring Integration
An Unreal® Engine project as proof-of-concept for receiving physiological data from Polar® H10 heart rate monitor. A conceivable application could be an exercise game or a physical eSports tournament like «Arena Games Triathlon»™ (cp. [13]).
Index Terms:
Technology:
Tags: UE, PolarH10, ECG, HR, HRM, PSL, ADB, BLE, USB, PubSub, MQTT, JSON, IOT, M2M
We implement a general data flow as shown in listing 1.1.
Listing 1.1.: General Data Flow
Data Producer —(MQTT)→ MQTT-Broker —(MQTT)→ MQTT-Client
We use system components as follows (for the specific data flow see Listing 1.2.):
Listing 1.2.: Specific Data Flow
Polar H10 –(Polar BLE SDK)→ Polar Sensor Logger –(MQTT)→ Mosquitto –(MQTT)→ Unreal Engine IoT-Plugin MQTT
The following shows the setup in reverse order of the data flow: Unreal Engine and Mosquitto on Windows—where we furthermore configure the firewall, use Wireshark and Android Debug Bridge, on Android we setup Polar Sensor Logger.
MQTT standard port is 1883, we will use TCP as transport. In the Windows Defender Firewall we allow TCP port 1883, e.g., by using an administrative PowerShell (see listing 2.1.).
Listing 2.1.: Firewall Rule "Allow TCP Port 1883"
New-NetFirewallRule -DisplayName "Allow TCP Port 1883" -Direction inbound -Profile Any -Action Allow -LocalPort 1883 -Protocol TCP
We make use of Wireshark to monitor the MQTT messages sent over port 1883 (cp. [4] and [5]).
Listing 2.2.: Use of Chocolatey to Install Wireshark
choco install wireshark
Listing 2.3.: Wireshark Filter TCP Port 1883
tcp.port == 1883
Figure 2.1.: Wireshark Dissecting Port 1883
Clone UE project "Heartbeat" using git, e.g., by git clone https://github.com/brugr9/Heartbeat51.git
and startup the project.
In the UE Heartbeat project we use the plugin "Built-In > IOT > MQTT" (see Figure 2.2.1.). Note: As of UE 5.1, the plugin is beta and not yet documented.
Figure 2.2.1.: Unreal Engine Plugins Browser Tab with Built-in IOT Plugin "MQTT"
In the "Project Settings > Plugin > MQTT"
localhost
, Port 1883
and Scheme MQTT
.10
messages per second. (cp. figure 2.2.2.).Figure 2.2.2.: Unreal Engine Project Settings, Plugins - MQTT
Map Map_PSL_Demo
holds a Blueprint BP_PSL_Demo
instance and additionally a TextRenderActor
instance, which is assigned to the BP_PSL_Demo
variable TextRender
as Object Reference (see figure 2.3.).
Figure 2.3.: Map_PSL_Demo with instances of Blueprint BP_PSL_Demo and TextRenderActor in the Outliner and in the Viewport
Blueprint BP_PSL_Demo
has components and variables as follows (see figure 2.4.):
HeartMesh
HeartbeatTimeline
(see figure 2.5.):PulseOut
1.00
on
MyTopic
, default value set to psl/hr
MyClient
MySubscription
TextRender
(public)TextRenderVisibilityTimer
Figure 2.4.: Blueprint BP_PSL_Demo, Variable TextRender
Figure 2.5.: Blueprint BP_PSL_Demo, Timeline Component HeartbeatTimeline
Blueprint BP_PSL_Demo
has events as follows (see figure 2.6.):
EventBeginPlay
, EventEndPlay
OnConnect
, OnDisconnect
, OnMessage
HeartbeatStandby
, HeartbeatUpdate
, HeartbeatDeactivate
TextRenderBlink
, HeartbeatReset
Testing01
, Testing02
, Testing03
Blueprint BP_PSL_Demo
has Event Graph sections as follows (see figure 2.6.):
Figure 2.6.: Blueprint BP_PSL_Demo, Event Graph Overview
Install Mosquitto MQTT-Broker (cp. [6]) and start the Windows Service "Mosquitto Broker" (see figure 2.9.).
Figure 2.9.: Mosquitto Broker as Windows Service
On the Android device enable USB Debugging mode (cp. [7]):
- Launch the
Settings
application.- Tap the
About Phone
option (generally found near the bottom of the list).- Then tap the
Build Number
option 7 times to enable Developer Mode. You will see a toast message when it is done.- Now go back to the main
Settings
screen and you should see a newDeveloper Options
menu you can access.- Go in there and enable the
USB Debugging
mode option.- Connect the Android device to the PC by USB cable.
On the PC using an administrative PowerShell setup "Android Debug Bridge" ADB (cp. [7]):
Back on the Andorid, a prompt "Allow USB Debugging" is shown, accept by hitting OK
Listing 2.4.: Use of Chocolatey to Install Android Debug Bridge
choco install adb
Listing 2.5.: Android Debug Bridge Startup
adb reverse tcp:1883 tcp:1883
Mount the Polar H10 sensor on the chest strap and wear the same. On the Android device ...
ECG
solely (cp. figure 2.10.).MQTT
solely (cp. figure 2.10.).
127.0.0.1
1883
psl
MyPSL-01
OK
SEEK SENSOR
Polar H10 12345678
(ID will differ) (cp. figure 2.12.)OK
Figure 2.10.: PSL, Main Tab | Figure 2.11.: PSL, Dialogue "MQTT Settings" | Figure 2.12.: PSL, Dialogue "Seek Sensor" | Figure 2.13.: PSL, Main Tab, Connected |
With Polar Sensor Logger main tab entry "SDK data select" option ECG activated, PSL publishes two topics:
psl/ecg
(cp. listing 2.6.): delivers a JSON-Object as payload containing, among others
"ecg": [ ... ]
, a JSON-Array of ECG values in microvolts [uV]psl/hr
(cp. listing 2.7.): delivers a JSON-Object as payload containing, among others
"hr": 64
containing a JSON-Integer which corresponds to the heart rate in beats per minute (bpm)"rr": [ 938 ]
containing a JSON-Array of RR intervals in milliseconds [ms]We will consume the latter topic psl/hr
only.
Listing 2.6.: Topic psl/ecg, example JSON-Object Payload
{
"clientId": "MyPSL-01",
"deviceId": "12345678",
"sessionId": 1234567890,
"sampleRate": 130,
"timeStamp": 1234567890123,
"sensorTimeStamp": 123456789012345678,
"ecg": [
-91,
-91,
-103,
-117,
...
]
}
Listing 2.7.: Topic psl/hr, example JSON-Object Payload
{
"clientId": "MyPSL-01",
"deviceId": "12345678",
"sessionId": 1234567890,
"timeStamp": 1234567890123,
"hr": 64,
"rr": [
938
]
}
In Unreal Editor with Level Map_PSL_Demo
open, click the Play
button ► in the level editor to start Play-in-Editor (PIE) (see listing 3.1.).
Listing 3.1.: Output Log of Map_PSL_Demo starting PIE
[...]
LogWorld: Bringing World /Game/UEDPIE_0_Map_PSL_Demo.Map_PSL_Demo up for play (max tick rate 0)
LogWorld: Bringing up level for play took: 0.000950
LogOnline: OSS: Created online subsystem instance for: :Context_6
[...]
PIE: Server logged in
PIE: Play in editor total start time 0.132 seconds.
[...]
On EventBeginPlay
an MQTT-Client is created and connected (see figure 3.1.). The MQTT-Plugin writes to the output log with custom log category LogMQTTCore
(see listing 3.2.). Wireshark dissecting port 1883 lists, e.g., the Connect Command
sent from the UE MQTT-Client (see figure 3.2.).
With event OnConnect
– if the connection was accepted – the topic is subscribed and event HeartbeatStandby
is called (cp. section 3.1.1.). With event OnMessage
the received message is evaluated by calling event HeartbeatUpdate
(cp. section 3.1.2.).
Figure 3.1.: Blueprint BP_PSL_Demo, Event Graph Section 'Startup Messaging'
Listing 3.2.: Output Log of Map_PSL_Demo starting PIE
[...]
LogMQTTCore: VeryVerbose: Created MQTTConnection for 127.0.0.1
LogMQTTCore: Display: Created new Client, Num: 1
LogMQTTCore: Verbose: Set State to: Connecting
LogMQTTCore: Verbose: Queued Subscribe message with PacketId 1., and Topic Filter: 'psl/hr'
[...]
LogMQTTCore: Verbose: Copy outgoing operations to buffer
LogMQTTCore: Verbose: Operations deferred: 2
LogMQTTCore: Verbose: Processing incoming packets of size: 4
LogMQTTCore: Verbose: Set State to: Connected
LogMQTTCore: VeryVerbose: Handled ConnectAck message.
LogMQTTCore: Verbose: Copy outgoing operations to buffer
LogMQTTCore: Verbose: Operations deferred: 0
LogMQTTCore: Verbose: Processing incoming packets of size: 5
LogMQTTCore: VeryVerbose: Handled SubscribeAck message with PacketId 1.
LogMQTTCore: Verbose: Processing incoming packets of size: 2
LogMQTTCore: VeryVerbose: Handled PingResponse message.
LogMQTTCore: Verbose: Copy outgoing operations to buffer
LogMQTTCore: Verbose: Operations deferred: 0
LogMQTTCore: Verbose: Copy outgoing operations to buffer
LogMQTTCore: Verbose: Operations deferred: 0
[...]
Figure 3.2.: Wireshark Dissecting Port 1883, Connect Command from Unreal Engine MQTT Client Instance
With event OnConnect
the topic is subscribed and event HeartbeatStandby
is called, which starts a visual feedback (see figure 3.3.): The RotatingMovement
is activated and the TextRenderVisibilityTimer
is started looping
within 0.5
seconds calling event TextRenderBlink
which switches the TextRender
visibility on and off by a FlipFlop
.
As result the Mesh Component HeartMesh
rotates and the TextRenderActor TextRender
is blinking (see figures 3.4.1. and 3.4.2.).
Figure 3.3.: Blueprint BP_PSL_Demo, Event Graph Section 'Heartbeat Standby'
Figure 3.4.1.: Screenshots of Map_PSL_Demo PIE, Heartbeat Standby
Figure 3.4.2.: Animation Screenshot of Map_PSL_Demo PIE, Heartbeat Standby
With event OnMessage
the received message payload is evaluated by calling event HeartbeatUpdate
, which updates the visual feedback (see figure 3.5.): The RR interval from JSON-Field rr
in Milliseconds [ms] is converted to [Hz] and is set as HeartbeatTimeline
play rate. The HR value from JSON-Field hr
is set to TextRender
.
As result the Mesh-Component HeartMesh
bumps frequently as given by RR interval and the TextRenderActor TextRender
shows the heart rate (see figures 3.6.1. and 3.6.2.).
Figure 3.5.: Blueprint BP_PSL_Demo, Event Graph Section 'Heartbeat Update'
Figure 3.6.1.: Screenshots of Map_PSL_Demo PIE, Heartbeat Update
Figure 3.6.2.: Animation Screenshot of Map_PSL_Demo PIE, Heartbeat Update
With event OnMessage
the received message payload is printed to output log (see listing 3.3.).
Listing 3.3.: Output Log of Map_PSL_Demo running PIE and logging the received Payloads
[...]
LogMQTTCore: Verbose: Processing incoming packets of size: 157
LogMQTTCore: VeryVerbose: Handled Publish message.
LogBlueprintUserMessages: [BP_PSL_Demo_C_1] {
"clientId": "MyPSL-01",
"deviceId": "12345678",
"sessionId": 1234567890,
"timeStamp": 1234567890123,
"hr": 64,
"rr": [
938
]
LogMQTTCore: Verbose: Processing incoming packets of size: 157
LogMQTTCore: VeryVerbose: Handled Publish message.
LogBlueprintUserMessages: [BP_PSL_Demo_C_1] {
"clientId": "MyPSL-01",
"deviceId": "12345678",
"sessionId": 1234567890,
"timeStamp": 1235678901234,
"hr": 124,
"rr": [
484
]
[...]
With stopping PIE, on EventEndPlay
the topic is unsubscribed (see figure 3.7.). With event HeartbeatDeactivate
the Mesh Component HeartMesh
and the TextRenderActor animation is stopped and reset (see figure 3.8.). Then the MQTT-Client disconnects, OnDisconnect
a message is printed to the output log (see listing 3.4.).
Figure 3.7.: Blueprint BP_PSL_Demo, Event Graph Section 'Teardown'
Figure 3.8.: Blueprint BP_PSL_Demo, Event Graph Section 'Heartbeat Deactivate'
Listing 3.4.: Output Log of Map_PSL_Demo stopping PIE
[...]
LogWorld: BeginTearingDown for /Game/UEDPIE_0_Map_PSL_Demo
LogMQTTCore: Verbose: Set State to: Disconnecting
[...]
LogMQTTCore: Verbose: Copy outgoing operations to buffer
LogMQTTCore: Verbose: Operations deferred: 1
LogMQTTCore: Verbose: Set State to: Disconnected
[...]
LogMQTTCore: Verbose: Set State to: Stopping
LogMQTTCore: Verbose: Abandoning Operations
LogMQTTCore: Verbose: Abandoning Operations
LogMQTTCore: VeryVerbose: Destroyed MQTTConnection at 127.0.0.1
[...]
The Quality of Service (QoS) level is an agreement between the sender of a message and the receiver of a message that defines the guarantee of delivery for a specific message. There are 3 QoS levels in MQTT:
- At most once (0)
- At least once (1)
- Exactly once (2)
When you talk about QoS in MQTT, you need to consider the two sides of message delivery:
- Message delivery form the publishing client to the broker.
- Message delivery from the broker to the subscribing client.
We will look at the two sides of the message delivery separately because there are subtle differences between the two. The client that publishes the message to the broker defines the QoS level of the message when it sends the message to the broker. The broker transmits this message to subscribing clients using the QoS level that each subscribing client defines during the subscription process. If the subscribing client defines a lower QoS than the publishing client, the broker transmits the message with the lower quality of service. (HiveMQ, cp. [9.1])
A retained message is a normal MQTT message with the retained flag set to true. The broker stores the last retained message and the corresponding QoS for that topic. Each client that subscribes to a topic pattern that matches the topic of the retained message receives the retained message immediately after they subscribe. The broker stores only one retained message per topic. (HiveMQ, cp. [9.2])
In a healthy person, the heart does not beat with a fixed frequency, i.e. with a resting pulse of, for example, 60 heartbeats per minute, each beat does not occur after exactly one second or 1000 milliseconds. Fluctuations of 30 to 100 milliseconds in the heartbeat sequence occur as a natural mode of operation of the heart.
Heart rate variability (HRV) is the amount by which the time interval between successive heartbeats (interbeat interval, IBI) varies from beat to beat. The magnitude of this variability is small (measured in milliseconds), and therefore, assessment of HRV requires specialized measurement devices and accurate analysis tools. Typically HRV is extracted from an electrocardiogram (ECG) measurement by measuring the time intervals between successive heartbeats [...]. Heart rate variability in healthy individuals is strongest during rest, whereas during stress and physical activity HRV is decreased. The magnitude of heart rate variability is different between individuals. High HRV is commonly linked to young age, good physical fitness, and good overall health. (Kubios, cp. [10]).
The RR interval RRI is an interbeat interval IBI, more precisely the time elapsed between two successive R-waves of the QRS signal on the electrocardiogram, in milliseconds [ms] (cp. [11] and [12]).
This documentation has not been reviewed or approved by the Food and Drug Administration FDA or by any other agency. It is the users responsibility to ensure compliance with applicable rules and regulations—be it in the US or elsewhere.
To acknowledge this work, please cite
Bruggmann, R. (2023): Unreal® Engine Project "Heartbeat" [Computer software], Version v5.1.0. Licensed under Creative Commons Attribution-ShareAlike 4.0 International. Online: https://github.com/brugr9/heartbeat51
@software{Bruggmann_Heartbeat_2023,
author = {Bruggmann, Roland},
year = {2023},
version = {v5.1.0},
title = {{Unreal Engine Project 'Heartbeat'}},
url = {https://github.com/brugr9/heartbeat51}
}
Unreal® Engine Project "Heartbeat" © 2023 by Roland Bruggmann is licensed under Creative Commons Attribution-ShareAlike 4.0 International