WARNING: The Openvidu Load Test project is having some big changes in functionality and architecture, so the latest versions in the repository are not stable.
What you can do is clone the repository on the commit of one of the previous stable releases (or use the latest tags in the Releases):
- KMS if you want to use emulation of browsers based on KMS.
- Node if you want to use emulation of browsers based on Node webrtc (both versions work for real browsers).
After that, you will have to download the latest openvidu-browser (or the version of openvidu-browser you are using in your OpenVidu deployment) and add it to the directory at browser-emulator/public, and modify line 6 of browser-emulator/public/index.html to use the latest openvidu-browser.
This repository contains a distributed tool that allows you to perform a load test against an OpenVidu CE or an OpenVidu PRO deployment.
Take into account that you must to deploy OpenVidu platform before using this tool.
Browser-emulator: Worker service implemented in NodeJS and controlled with a REST protocol that capable of connecting to an OpenVidu session and sending and receiving WebRTC media using openvidu-browser library. It is able to start and launch Chrome/Firefox browsers using Selenium, emulating a fully real user.
Loadtest Controller: Controller service in charge of the coordination of the browser-emulator workers. It reads the load test scenario from a file and control the browser-emulator workers to connect participants loading OpenVidu platform.
In the machine you want to execute the controller you need to have Java 11 platform installed.
Then follow these steps:
git clone https://github.com/OpenVidu/openvidu-loadtest.git
Running the following command under project root directory you will be able to edit the load-test/src/main/resources/application.properties
which contains a list of configuration parameters that you will can customize.
nano loadtest-controller/src/main/resources/application.properties
In this file you will see:
# Load Test Parameters (Required)
OPENVIDU_URL=https://openvidu_pro_url
OPENVIDU_SECRET=openvidu_pro_secret
SESSION_NAME_PREFIX=LoadTestSession
USER_NAME_PREFIX =User
SECONDS_TO_WAIT_BETWEEN_PARTICIPANTS=1
SECONDS_TO_WAIT_BETWEEN_SESSIONS=2
SECONDS_TO_WAIT_BEFORE_TEST_FINISHED=10
SECONDS_TO_WAIT_BETWEEN_TEST_CASES=10
# For testing locally, fill it with the worker ip address: 195.166.0.0
WORKER_URL_LIST=
General AWS parameters
# For testing on AWS
# For launching EC2 instances and allowing to workers upload recordings to S3 bucket
AWS_ACCESS_KEY=your-aws-key
AWS_SECRET_ACCESS_KEY=your-secret-key
WORKER_AMI_ID=ami-xxxxxxxxxxx
# We recommend c5 type. https://aws.amazon.com/ec2/instance-types/
WORKER_INSTANCE_TYPE=c5.xlarge
# By default, the browser-emulator service is listening on:
# 5000 (REST API)
# 5001 (WebSocket)
# The SG will need these ports opened.
WORKER_SECURITY_GROUP_ID=sg-xxxxxxxxx
WORKER_INSTANCE_REGION=
# Numbers of workers to launch before the test starts
WORKERS_NUMBER_AT_THE_BEGINNING=1
# Overcoming this CPU percentage threshold a new worker will be launched
WORKER_MAX_LOAD=80
# Number of new workers incrementation
WORKERS_RAMP_UP=1
# Boolean param for MANUAL participants allocation
MANUAL_PARTICIPANTS_ALLOCATION=false
# Number of participants per worker
USERS_PER_WORKER=
################################################
#### MinIO credentials and information if using it to save recordings instead of S3
################################################
# MinIO Endpoint
MINIO_HOST=
# MinIO Port (default 443)
MINIO_PORT=443
# MinIO Access Key
MINIO_ACCESS_KEY=
# MinIO Secret Key
MINIO_SECRET_KEY=
# MinIO Bucket Name
MINIO_BUCKET=
These properties allow record a participant (launching an external chrome browser) for check the received video quality.
When test finishes, it will saved in the S3 Bucket.
# For video quality control (enabled for AWS testing only)
# It'll start a new ec2 instance where a new participant will be connected using a real Chrome Browser
# it will start to record the session when media node has archieved this value
# Needs an ElasticSearch instance to work
MEDIANODE_LOAD_FOR_START_RECORDING=0
# it will start to record the sessions gruped by the specifyed value.
# 0: Recording disabled, 1 recording starts each session, 2 recording starts each two session ...
RECORDING_SESSION_GRUPED_BY=0
# Bucket name where the recordings will be saved
# Record each MediaStream in each worker for Quality of Experience analysis
QOE_ANALYSIS_RECORDINGS=true
# Perform qoe analysis in the same worker as they were recorded in, if false the recordings will only be uploaded to S3
QOE_ANALYSIS_IN_SITU=false
# Video information needed for in situ QoE analysis, read https://github.com/OpenVidu/openvidu-loadtest/tree/master/browser-emulator#running-qoe-analysis for more information
VIDEO_PADDING_DURATION=1
VIDEO_FRAGMENT_DURATION=5
S3_BUCKET_NAME=openvidu-loadtest-capacity
These properties allow retry the participant creation when it fails
RETRY_MODE=false
RETRY_TIMES=5
These properties allow you to select the video to use when the browsers used are real browsers. There are 3 default options and you can also use a custom video, check the description of the browser-emulator Initialize instance API for more information about these options:
# Video type options: BUNNY, INTERVIEW, GAME, CUSTOM
VIDEO_TYPE=BUNNY
VIDEO_WIDTH=640
VIDEO_HEIGHT=480
VIDEO_FPS=30
# If VIDEO_TYPE is CUSTOM, you have to specify the video and audio URLs
VIDEO_URL=
AUDIO_URL=
WARNING: The experiments we made have been tested with Elasticsearch 7.8.0 version. A deployment with a different version could cause errors.
# Monitoring Parameters (Optional)
ELASTICSEARCH_USERNAME=elasticusername
ELASTICSEARCH_PASSWORD=password
KIBANA_HOST=https://kibanahost
| Make sure your OpenVidu Configuration has the following properties for a good monitoring:
OPENVIDU_PRO_STATS_MONITORING_INTERVAL=1
OPENVIDU_PRO_STATS_WEBRTC_INTERVAL=1
Recommended to leave default values unless you know what you are doing.
# Insert the users in batches or one by one
BATCHES=true
# Maximum number of concurrent requests. If left blank, defaults to number of available processors + 1.
BATCHES_MAX_REQUESTS=10
# Wait after user or batch insertion
WAIT_COMPLETE=true
# Install VNC server on workers for debug purposes
DEBUG_VNC=false
Controller is being developed to allow the configuration of the load test cases: number of participants, typology, number of sessions, etc.
Currently, the only available option is the change of the number of participants of the session.
To configure the test cases the file loadtest-controller/src/main/resources/test_cases.json
has to be edited:
{
"testcases": [
{
"typology": "N:N",
"participants": ["2", "3", "5"],
"sessions": "20",
"browserMode": "EMULATE",
"frameRate": "30",
"resolution": "640x480",
"openviduRecordingMode": "",
"desciption": "This test case will add infinite sessions (until it reaches its limit) of publishers that the array of participants indicates"
},
{
"typology": "N:M",
"participants": ["1:10", "1:100", "2:10", "2:30", "2:50", "3:10", "3:30", "3:50"],
"sessions": "infinite",
"desciption": "This test case will add infinite sessions (until it reaches its limit) with as many PUBLISHERS and SUBSCRIBERS as the participants array indicates."
},
{
"typology": "TEACHING",
"participants": ["2:10", "2:30", "2:50", "3:10", "3:30", "3:50"],
"sessions": "infinite",
"desciption": "This test case will add infinite sessions (until it reaches its limit) with as many PUBLISHERS (teachers:students) as the participants array indicates. The students (FAKE SUBSCRIBERS, they will be PUBLIHSERS with only audio) will only publish audio"
},
{
"typology": "TERMINATE",
"desciption": "This terminate all EC2 instances launched for loadtest purposes"
}
]
}
Properties | Type | Description |
---|---|---|
typology * | N:N , N:M , TEACHING or TERMINATE |
N:N: All users will be PUBLISHERS and SUBSCRIBERS N:M: N number will be PUBLISHERS and M number will be SUBSCRIBERS TEACHING: It will emulate a teaching videoconference. The students (FAKE SUBSCRIBERS, they will be PUBLIHSERS with only audio) will only publish audio TERMINATE: It will terminate all EC2 instances launched for loadtest purposes |
participants * | String [] | Number of participants in each session. It will run the same test case as many time as the positions in the array. For example: For N:N typology: ["2","5"] where all of them will be publishers and test case will be run twice (sessions with 2 and 5 participants) For N:M typology: ["2:30"] where 2 will be publishers and 30 will be subscribers For TEACHING typology: ["2:30"] where 2 will be teachers and 30 will be students |
sessions * | infinite or Number |
infinite: It will create infinite sessions until the limit is reached. Number value: It will create only (number value) sessions |
browserMode * | EMULATE or REAL |
EMULATE: the service will emulate a browser. REAL: the service will launch a Chrome browser docker container. Choosing EMULATE , you must ensure that OpenVidu aren't forcing H264 coded |
resolution | String | Desired resolution of the video. 640x480 or 1280x720 Default 640x480 |
frameRate | Number (0-30) | Desired framerate of the video in frames per second. Default 30 |
openviduRecordingMode | String | COMPOSED or INDIVIDUAL See OpenVidu recording. |
browserRecording | Boolean | If browserMode is REAL and you want record the Chrome browser using ffmpeg. Otherwise, If browserMode is EMULATE and you have started browser-emulator with KMS user type (see worker running options) Default false . |
showBrowserVideoElements | Boolean | If browserMode is REAL and you want show videos elements into the app running in Chrome. Default true |
headlessBrowser | Boolean | If browserMode is REAL and you want launch a headless Chrome. Default false . See Headless Chromium |
When you execute the load test the controller will create sessions and will connect participants into them automatically.
The load test has several stop conditions. It will stop when any of the following conditions occur:
When loadtest-controller receives an exception error from any of the workers.
When any of workers can't create a new participant because of lack of resources or a long time trying creating it.
Take into account that errors can be produced by OpenVidu errors or if the worker itself is overloaded. Please control CPU usage in workers.
When an error in a worker is produced, the load test will stop and will order to destroy and close every participants and sessions to the workers.
For start with it, run the following command:
cd loadtest-controller
mvn spring-boot:run
mvn spring-boot:run -Dspring-boot.run.jvmArguments="-DTERMINATE_WORKERS=true"
For illustration proporses here is an example composed by 2 workers and the load test app all of them running locally.
As you can see in the load test and workers logs, the load test app will create sessions and will add participants into them following a round-robin scheduling.
Load test logs
Which it means that, for this sample with sessions of 2 participans, each participant will be create by one different worker. The User0
will be create by the worker 1 and the User1
will be created by the worker2.
Thus achieving more capacity in the load test and less resource consumption.
Worker 1 logs
Worker 2 logs
The loadtest-controller will create a report result on the root directory /openvidu-loadtest/loadtest-controller
with the name result.txt. This file will contains the following information:
----- Test Case Report Fri Apr 30 12:39:46 CEST 2021 -----
Browser approach:
Browser with recording:
Session typology:
Participants per session:
Number of sessions created:
Number of sessions completed:
Number of participants created:
Number of workers used:
Stop reason:
Test duration:
Kibana url:
If you're testing OpenVidu PRO, the loadtest-controller will import a Kibana Dashboard automatically at the beginning of the test. This dashboard will include Kibana Visualizations with all metrics retrieved from OpenVidu.
To allow that the Load Test App import it, you must fill the monitoring parameters.
What happen if the dashboard have not been imported automatically?
An alternative if the dashboard is not imported automatically is import it manually. You just to follow these steps:
Management > Stack Management
.Saved Objects
option, under Kibana section.Besides, if you have deployed OpenVidu PRO you can create your own visualizations in Kibana or you can export the raw data and use another tool. Remember that ELK stack has monitoring information about the platform (CPU, memory usage, bandwidth, etc) and also high level information (sessions, participants, etc.)
This service provides a simple REST API that will be used by Load Test application and it allows:
Ping to instance. Do ping to check if instance is ready.
Initialize instance. Initialize monitoring stuffs like ElasticSearch env variables and Metricbeat container. This request also set the AWS public and secret keys for uploading the video recordings files to S3 (just in case the test case includes recording).
Create a participant (PUBLISHER
or SUBSCRIBER
) using a custom token created by you or creating a new token.
Delete a specific participant by its connectionId
Delete all participant with a specific role (PUBLISHER
or SUBSCRIBER
).
Delete all participant
This services also is listening for a WebSocket communication on ws:browser-emulator-addres:5001/events
. It will send information from openvidu-browser to the loadtest-controller.
See browser-emulator docummentation for more info.
This work has been supported by the Regional Government of Madrid (Spain) (CM) through project EDGEDATA-CM (P2018/TCS-4499) cofunded by FSE & FEDER.