JMeter k8s starterkit

This is a template repository from which you can start load testing faster when injecting load from a kubernetes cluster.

You will find inside it the necessary to organize and run your performance scenario. There is also a node monitoring tool which will monitor all your injection nodes. As well an embeded live monitoring with InfluxDB and Grafana

Thanks to Kubernauts for the inspiration !

Features

Feature	Supported	Comment
Flexibility at run time	Yes	With .env file (threads, duration, host)
Distributed testing	Yes	Virtually unlimited with auto-scaling
JMeter Plugin support	Yes	Modules are installed at run time by scanning the JMX needs
JMeter Module support	Yes	JMeter include controller are supported if path is just the name of the file in the Include Controler
JMeter CSV support	Yes	CSV files are splitted prior to launch the test and unique pieces copied to each pods, in the JMeter scenario, just put the name of the file in the path field
Node auto-scaling	Yes	By requesting ressources at deployment time, the cluster will scale automatically if needed
Reporting	Yes	The JMeter report is generated at the end of the test inside the master pod if the -r flag is used in the start_test.sh
Live monitoring	Yes	An InfluxDB instance and a Grafana are available in the stack
Report persistance	Yes	A persistence volume is used to store the reports and results
Injector nodes monitoring	Yes	Even if autoscaling, a Daemon Set will deploy a telegraf instance and persist the monitoring data to InfluxDB. A board is available in Grafana to show the Telegraf monitoring
Multi thread group support	Not really	You can add multi thread groups, but if you want to use JMeter properties (like threads etc..) you need to add them in the .env and update the start_test.sh to update the "user_param" variable to add the desired variables
Mocking service	Yes	A deployment of Wiremock is done inside the cluster, the mappings are done inside the wiremock configmap. Also an horizontal pod autoscaler have been added
JVM Monitoring	Yes	JMeter and Wiremock are both Java application. They have been packaged with Jolokia and Telegraf and are monitored
Pre built Grafana Dashboards	Yes	4 Grafana dashboards are shipped with the starter kit. Node monitoring, Kubernetes ressources monitoring, JVM monitoring and JMeter result dashboard.
Ressource friendly	Yes	JMeter is deployed as batch job inside the cluster. Thus at the end of the execution, pods are deleted and ressources freed

Getting started

Prerequisites :

A kubernetes cluster (of course) (amd64 and arm64 architecture are supported)
kubectl installed and a usable context to work with
(Optionnal) A JMeter scenario (the default one attack Google.com)

1. Preparing the repository

You need to put your JMeter project inside the scenario folder, inside a folder named after the JMX (without the extension). Put your CSV file inside the dataset folder, child of scenario Put your JMeter modules (include controlers) inside the module folder, child of scenario

datasetand moduleare in scenario and not below inside the <project> folder because, in some cases, you can have multiple JMeter projects that are sharing the JMeter modules (that's the goal of using modules after all).

Below a visual representation of the file structure

+-- scenario
|   +-- dataset
|   +-- module
|   +-- my-scenario
|       +-- my-scenario.jmx
|       +-- .env

2. Deploying the Stack

If you want to deploy in AWS, please refer to aws folder

From this repository

kubectl create -R -f k8s/

This will deploy all the needed applications :

JMeter master and slaves
Telegraf operator to automatically monitor the specified applications
Telegraf as a DaemonSet on all the nodes
InfluxDB to store the date (with a 5GB volume in a PVC)
Grafana with a LB services and 4 included dashboard
Wiremock

Using helm

This helm project is at an very early stage, feel free to test it and open any issue for any feedbacks. Thanks you

helm repo add jmeter-k8s-starterkit-helm-charts https://rbillon59.github.io/jmeter-k8s-starterkit-helm-chart/
helm install jmeter-k8s-starterkit-helm-charts/jmeter-k8s-starterkit --generate-name

3. Starting the test

./start_test.sh -j my-scenario.jmx -n default -c -m -i 20 -r

Usage :

   -j <filename.jmx>
   -n <namespace>
   -c flag to split and copy csv if you use csv in your test
   -m flag to copy fragmented jmx present in scenario/project/module if you use include controller and external test fragment
   -i <injectorNumber> to scale slaves pods to the desired number of JMeter injectors
   -r flag to enable report generation at the end of the test

The script will :

Delete and create again the JMeter jobs.
Scale the JMeter slave deployment to the desired number of injectors
Wait to all the slaves pods to be available. Here, available means that the filesystem is reacheable (liveness probe that cat a file inside the fs)
If needed will split the CSV locally then copy them inside the slave pods
If needed will upload the JMeter modules inside the slave pods
Send the JMX file to each slave pods
Generate and send a shell script to the slaves pods to download the necessary plugins and launch the JMeter server.
Send the JMX to the controller
Generate a shell script and send it to the controller to wait for all pods to have their JMeter slave port listening (TCP 1099) and launch the performance test.

4. Gethering results from the master pod

After the test have been executed, the master pod job is in completed state and then, is deleted by the cleaner cronjob.

To be able to get your result, a jmeter master pod must be in running state (because the pod is mounting the persistantVolume with the reports inside).

The master pod default behaviour is to wait until the load_test script is present in the pod

You can run

# If a master pod is not available, create one
kubectl apply -f k8s/jmeter/jmeter-master.yaml
# Wait for the pod is Running, then
kubectl cp -n <namespace> <master-pod-id>:/report/<result> ${PWD}/<local-result-name>
# To copy the content of the report from the pod to your local

You can do this for the generated report and the JTL for example.

Rbillon59 / jmeter-k8s-starterkit

readme