beinfluential88
commented
2 years ago Procedure to implement Gatling
Objective: With the aim of monitoring the performance of our system and detecting imminent system failures preemptively, we are planning to implement the following tools for system analysis purposes.
- Gatling: A load-testing tool that mimics the behavior of a user. (Heavier load can be applied to the system by increasing the number of synthetic users)
- Grafana: A dashboard tool that can be used to real-time monitoring of the system performance. Grafana queries Prometheus to retrieve relevant metrics from Prometheus, and these queries can be adjusted for different analysis.
- Prometheus: A system for gathering and storing metrics as a time-series database.
Pre-requisite
-
AWS EKS cluster should be up and running which will serve the 4 microservices, db (database service), s1 (user service), s2 (music service), and s3 (playlist service). a. Starting a fresh EKS cluster.
make -f eks.mak start -
Ensure AWS DynamoDB is initialized. The tables have to be available for db (database service) to serve the other 3 microservices. (s1/s2/s3)
aws dynamodb list-tables -
Provision the cluster. This includes; a. Create a namespace within the cluster in which applications will be placed.
kubectl create ns c756nskubectl config set-context --current --namespace=c756nsb. Provision the Kubernetes cluster. This includes “Installing Istio”, “Installing Prometheus stack by callingobs.makrecursively”, and “Deploying and monitoring the four microservices”.make -f k8s.mak provision -
Get the Grafana URL using which we can access the dashboard.
make -f k8s.mak grafana-url• User: admin • Password: prom-operator Note: The hostname is obtained from ‘istio” namespace.kubectl get -n istio-system svc/grafana-ingress -o jsonpath="{.status.loadBalancer.ingress[0]['ip','hostname']}"Parameters: 1: path to kubectl 2: namespace 3: the resource to query (typically an svc)
How does Gatling work?
1. We will be using a Gatling docker image which will allow us to create and apply synthetic load to our system.
ghcr.io/scp-2021-jan-cmpt-756/gatling:3.4.2
2. Scenarios of user behavior are defined in “ReadTables.scala”.
a. The original package name is defined, which will be used to trigger a Gatling instance later.
package proj756
b. The required imports for Gatling to work.
import scala.concurrent.duration._
import io.gatling.core.Predef._
import io.gatling.http.Predef._
c. “Utility” object
envVarToInt(“USERS”, 1) - Utility to get an Int from an environment variable. (e.g., The number of users is defined under “docker container run” command.)
envVar("CLUSTER_IP", "127.0.0.1") - Utility to get a string from an environment variable. (e.g., The cluster IP is defined under “docker container run” command.)
d. “RMusic” and “RUser” objects – Scenarios defined to be tested for respective services. Sending an HTTP get request with {UUID} continuously every one second.
Note: “eager()” loads the whole data in memory before the Simulation starts, saving disk access at runtime. “random()” randomly picks an entry in the sequence. “circular()” goes back to the top of the sequence once the end is reached.
e. “RUserVarying” and “RMusicVarying” objects – Scenarios defined to be tested for respective services. Sending a HTTP get request with {UUID} continuously with different intervals between calls. (Each interval is randomly selected between 1 and 60 seconds)
f. “ReadTablesSim” class – This class inherits properties from “Simulation” class and used to define HTTP protocols for simulations. (e.g., cluster IP is read from environment variables defined under “docker container run” command)
g. “ReadUserSim” and “ReadMusicSim” classes – These classes are directly called by “docker container run” command, which will inject independent users (as defined in “RMusic” and “RUser” objects) via HTTP protocols defined in “ReadTablesSim” class
h. “ReadBothVaryingSim” class – This class is directly called by “docker container run” command, which will inject concurrent users (as defined in “RMusicVarying” and “ReadUserVarying” objects) via HTTP protocols defined in “ReadTablesSim” class.
Note: There are 2 types of workload model for injection. - Open vs. Closed
• Closed systems, where you control the concurrent number of users. Closed system are system where the number of concurrent users is capped. At full capacity, a new user can effectively enter the system only once another exits.
• Open systems, where you control the arrival rate of users. Open systems have no control over the number of concurrent users: users keep on arriving even though applications have trouble serving them.
Note: For closed model, We have two methods that we use to inject users.
• constantConcurrentUsers(nbUsers).during(duration): Inject so that number of concurrent users in the system is constant
• rampConcurrentUsers(fromNbUsers).to(toNbUsers).during(duration): Inject so that number of concurrent users in the system ramps linearly from a number to another
3. Create a script that will trigger Gatling. (e.g., gatling-1-music.sh)
docker container run --detach --rm \
-v ${PWD}/gatling/results:/opt/gatling/results \
-v ${PWD}/gatling:/opt/gatling/user-files \
-v ${PWD}/gatling/target:/opt/gatling/target \
-e CLUSTER_IP=`tools/getip.sh kubectl istio-system svc/istio-ingressgateway` \
-e USERS=1 \
-e SIM_NAME=ReadMusicSim \
--label gatling \
ghcr.io/scp-2021-jan-cmpt-756/gatling:3.4.2 \
-s proj756.ReadMusicSim4. To list Gatling containers currently running
tools/list-gatling.sh
5. To stop all the Gatling containers.
tools/kill-gatling.sh
bingsoorim
In order for the team to come up with a concrete plan, we need to understand how Gatling is used for load testing and how Grafana and Prometheus allow us to monitor performance/load metrics.