Description

added InitContainer: gatling-waiter to runner pods to wait node scaling up before execution InitContainer: gatling-runner
- gatling-waiter execute infinite loop until specified parallelism numbers of pods starts gatling-waiter
  - other pod status are fetched from pods label gatling-waiter which is updated by gatling-waiter
- this mechanism can ensure executing gatling simulation at almost the same time (like in 10s)
made serviceAccountName required at Gatling CR and add sample manifest for service account, role, role binding
- because gatling-waiter requires permissions to execute kubectl commands

Prerequisite

runner pods must use serviceaccount with a permission for pod get, watch, list, patch

sample yaml for the service account

apiVersion: v1
kind: ServiceAccount
metadata:
  name: gatling-operator-worker
  namespace: gatling-system
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: gatling-system
  name: pod-reader
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "watch", "list", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: gatling-system
subjects:
  - kind: ServiceAccount
    name: gatling-operator-worker
    namespace: gatling-system
    apiGroup: ""
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: ""

Test

confirmed than serviceAccountName became required at Gatling CR
confirmed that gatling-waiter worked properly to wait node scaling.
confirmed that gatling simulation finished time difference between runner pods which ran on an already existed node and which ran on a scaled node is less than 10s.
confirmed that $ make kind-sample-deploy worked with new sample manifests

Execution Logs

#  serviceAccountName became required at Gatling CR
❯ k apply -f gatling.yaml
error: error validating "gatling.yaml": error validating data: ValidationError(Gatling.spec.podSpec): missing required field "serviceAccountName" in com.zozo.tech.gatling-operator.v1alpha1.Gatling.spec.podSpec; if you choose to ignore these errors, turn validation off with --validate=false

❯ k apply -f gatling.yaml
gatling.gatling-operator.tech.zozo.com/api-load-test created

# 2 of 5 runner pods are Pending due to insufficient resources
NAME                                                   READY   STATUS     RESTARTS   AGE   LABELS
gatling-operator-controller-manager-5dbddfc898-9mzdp   2/2     Running    0          25m   control-plane=controller-manager,pod-template-hash=5dbddfc898
api-load-test-runner-27w4d          0/1     Init:0/2   0          15s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-9r9k5          0/1     Init:0/2   0          15s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-dfl8z          0/1     Pending    0          15s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,job-name=api-load-test-runner
api-load-test-runner-lfvbs          0/1     Pending    0          15s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,job-name=api-load-test-runner
api-load-test-runner-svxjd          0/1     Init:0/2   0          15s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner

# waiting for nodes scaling up
NAME                                                  STATUS     ROLES    AGE     VERSION
gke-main-gatling-operator-controller--da347ed0-hnnr   Ready      <none>   4d23h   v1.21.5-gke.1302
gke-main-gatling-operator-worker-v1-76ebb351-47kz     Ready      <none>   6m52s   v1.21.5-gke.1302
gke-main-gatling-operator-worker-v1-cac12e8a-7ds8     NotReady   <none>   8s      v1.21.5-gke.1302
gke-main-gatling-operator-worker-v1-cac12e8a-pf5q     Ready      <none>   43m     v1.21.5-gke.1302
gke-main-gatling-operator-worker-v1-dc621851-s8jt     Ready      <none>   9m54s   v1.21.5-gke.1302
gke-main-gatling-operator-worker-v1-dc621851-zgss     NotReady   <none>   4s      v1.21.5-gke.1302

# 5 of 5 pods are scheduled and started InitContainer gatling-waiter after nod scaling up finished
NAME                                                   READY   STATUS     RESTARTS   AGE   LABELS
gatling-operator-controller-manager-5dbddfc898-9mzdp   2/2     Running    0          26m   control-plane=controller-manager,pod-template-hash=5dbddfc898
api-load-test-runner-27w4d          0/1     Init:0/2   0          93s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-9r9k5          0/1     Init:0/2   0          93s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-dfl8z          0/1     Init:0/2   0          93s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,job-name=api-load-test-runner
api-load-test-runner-lfvbs          0/1     Init:0/2   0          93s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,job-name=api-load-test-runner
api-load-test-runner-svxjd          0/1     Init:0/2   0          93s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner

# All pods gatling-waiter completed and started gatling-runner at almost the same time
NAME                                                   READY   STATUS     RESTARTS   AGE    LABELS
gatling-operator-controller-manager-5dbddfc898-9mzdp   2/2     Running    0          27m    control-plane=controller-manager,pod-template-hash=5dbddfc898
api-load-test-runner-27w4d          0/1     Init:1/2   0          2m5s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-9r9k5          0/1     Init:1/2   0          2m5s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-dfl8z          0/1     Init:1/2   0          2m5s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-lfvbs          0/1     Init:1/2   0          2m5s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-svxjd          0/1     Init:1/2   0          2m5s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner

# All runner pods and reporter pods completed
NAME                                                   READY   STATUS      RESTARTS   AGE     LABELS
gatling-operator-controller-manager-5dbddfc898-9mzdp   2/2     Running     0          30m     control-plane=controller-manager,pod-template-hash=5dbddfc898
api-load-test-reporter-kvs56        0/1     Completed   0          2m38s   controller-uid=fe312f10-9399-4825-b4c5-96cd84b9db58,job-name=api-load-test-reporter
api-load-test-runner-27w4d          0/1     Completed   0          5m34s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-9r9k5          0/1     Completed   0          5m34s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-dfl8z          0/1     Completed   0          5m34s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-lfvbs          0/1     Completed   0          5m34s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner
api-load-test-runner-svxjd          0/1     Completed   0          5m34s   controller-uid=bfda6c26-bd5f-4d24-9805-bc031afa6df9,gatling-waiter=initialized,job-name=api-load-test-runner

# confirmed that gatling simulation finished time difference between runner pods which ran on an already existed node and which ran on a scaled node is less than 10s
❯ k logs api-load-test-runner-27w4d -c gatling-runner
Wait until 2022-01-18 23:25:25
GATLING_HOME is set to /opt/gatling
Simulation Api started...

~~~

================================================================================
2022-01-18 23:26:00                                          20s elapsed
---- Requests ------------------------------------------------------------------
> Global                                                   (OK=2000   KO=0     )
> API                                   (OK=2000   KO=0     )
---- API (dev) ----------------------------------------------
[##########################################################################]100%
          waiting: 0      / active: 0      / done: 2000
================================================================================
Simulation Api completed in 20 seconds

❯ k logs api-load-test-runner-dfl8z -c gatling-runner
Wait until 2022-01-18 23:25:34
GATLING_HOME is set to /opt/gatling
Simulation Api started...

~~~

================================================================================
2022-01-18 23:26:09                                          20s elapsed
---- Requests ------------------------------------------------------------------
> Global                                                   (OK=2000   KO=0     )
> API                                   (OK=2000   KO=0     )
---- API (dev) ----------------------------------------------
[##########################################################################]100%
          waiting: 0      / active: 0      / done: 2000
================================================================================
Simulation Api completed in 20 seconds

❯ k logs api-load-test-runner-27w4d -c gatling-waiter
pod/api-load-test-runner-27w4d labeled
3/5 pods are ready
3/5 pods are ready
3/5 pods are ready
~~~
3/5 pods are ready
4/5 pods are ready
4/5 pods are ready
4/5 pods are ready
4/5 pods are ready
5/5 pods are ready

❯ make kind-sample-deploy
Cluster already exists
cd gatling && docker build -t gatling:20220121-102037 .
[+] Building 1.0s (13/13) FINISHED
 => [internal] load build definition from Dockerfile                                                                                       0.0s
 => => transferring dockerfile: 37B                                                                                                        0.0s
 => [internal] load .dockerignore                                                                                                          0.0s
 => => transferring context: 2B                                                                                                            0.0s
 => [internal] load metadata for docker.io/library/openjdk:8-jdk-alpine                                                                    0.9s
 => [1/8] FROM docker.io/library/openjdk:8-jdk-alpine@sha256:94792824df2df33402f201713f932b58cb9de94a0cd524164a0f2283343547b3              0.0s
 => [internal] load build context                                                                                                          0.0s
 => => transferring context: 296B                                                                                                          0.0s
 => CACHED [2/8] WORKDIR /opt                                                                                                              0.0s
 => CACHED [3/8] RUN mkdir -p gatling                                                                                                      0.0s
 => CACHED [4/8] RUN apk add --update wget bash libc6-compat &&   mkdir -p /tmp/downloads &&   wget -q -O /tmp/downloads/gatling-3.2.1.zi  0.0s
 => CACHED [5/8] WORKDIR  /opt/gatling                                                                                                     0.0s
 => CACHED [6/8] COPY user-files/simulations user-files/simulations                                                                        0.0s
 => CACHED [7/8] COPY user-files/resources user-files/resources                                                                            0.0s
 => CACHED [8/8] COPY conf/gatling.conf conf/gatling.conf                                                                                  0.0s
 => exporting to image                                                                                                                     0.0s
 => => exporting layers                                                                                                                    0.0s
 => => writing image sha256:3dfa03fa2944e219cb31a3db5dcb6000b13835c5e6400cbf72a7f386892e8403                                               0.0s
 => => naming to docker.io/library/gatling:20220121-102037                                                                                 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Loading sample image into kind
kind load docker-image gatling:20220121-102037 --name "gatling-cluster" -v 1
Image: "gatling:20220121-102037" with ID "sha256:3dfa03fa2944e219cb31a3db5dcb6000b13835c5e6400cbf72a7f386892e8403" not yet present on node "gatling-cluster-control-plane", loading...
Image: "gatling:20220121-102037" with ID "sha256:3dfa03fa2944e219cb31a3db5dcb6000b13835c5e6400cbf72a7f386892e8403" not yet present on node "gatling-cluster-worker", loading...
/Users/atsushi.kayama/Documents/codes/github.com/st-tech/gatling-operator/bin/kustomize build config/samples | sed -e "s,GATLING_IMAGE,gatling:20220121-102037,g" | kubectl apply -f -
serviceaccount/gatling-operator-worker unchanged
role.rbac.authorization.k8s.io/pod-reader unchanged
rolebinding.rbac.authorization.k8s.io/read-pods configured
secret/gatling-basic-simulation-secrets unchanged
secret/gatling-notification-slack-secrets unchanged
gatling.gatling-operator.tech.zozo.com/gatling-sample01 created

❯ k get po
NAME                                                   READY   STATUS    RESTARTS   AGE
gatling-operator-controller-manager-668ccc89cf-z5zd7   2/2     Running   0          3m47s
gatling-sample01-runner-9czrz                          1/1     Running   0          13s
gatling-sample01-runner-9gx2p                          1/1     Running   0          13s
gatling-sample01-runner-lxqgc                          1/1     Running   0          13s

❯ k get sa
NAME                                  SECRETS   AGE
default                               1         3m49s
gatling-operator-controller-manager   1         3m49s
gatling-operator-worker               1         2m37s

st-tech / gatling-operator

add gatling-waiter to wait node scaling before staring gatling simulation #31

Description

Prerequisite

Test