canonical / mysql-k8s-operator

A Charmed Operator for running MySQL on Kubernetes
https://charmhub.io/mysql-k8s
Apache License 2.0
8 stars 15 forks source link

[Performance] Mysql router reduces QPS #304

Open phvalguima opened 12 months ago

phvalguima commented 12 months ago

--- Testing Env --- Openstack Yoga Charmed Kubernetes 1.27/stable - bundle: https://pastebin.canonical.com/p/8BJpWhh2xk/ (m1.xlarge == 4vcpus / 16G RAM) Using localpath provisioner for storage: https://pastebin.canonical.com/p/g6K9XQ2fFV/ Mysql Bundle: https://pastebin.canonical.com/p/62mBxMWFHr/ # hostpath is defined as storage class in the bundle model-default.yaml: https://pastebin.canonical.com/p/zhvXyjzt9Y/ # For proxy setup

--- Setup Steps --- Source the openrc and add the openstack cloud (named here, pc1-openstack)

juju add-model mysql-k8s-cluster pc1-openstack --config ./config/model-default.yaml

Wait to finish deployment. Kubeconfig is available with: juju ssh kubernetes-control-plane/0 cat config > ~/.kube/config

Create the mysql k8s model:

juju add-k8s mysql-k8s-cluster # add the cluster as a cloud
juju add-model mysql mysql-k8s-cluster --config ./config/model-default.yaml
juju create-storage-pool hostpath kubernetes storage-class=microk8s-hostpath storage-provisioner=[microk8s.io/hostpath](http://microk8s.io/hostpath)
juju deploy <mysql-bundle>

The environment is deployed with a single Mysql (to force primary to unit 0) and 3x mysql routers. Deploy the extra units manually:

juju add-unit mysql-k8s --num-units 2

Wait for the units to settle.

Scale the test-app and start the continuous writes:

juju add-unit mysql-test-app --num-units 3
# Once the units are finished deployed, deploy for each unit:
juju run mysql-test-app/YYY start-continuous-writes

--- Connect to the environment --- To connect directly to the mysql primary pod, it is recommended to use sshuttle and expose the pod-to-pod network directly:

kubectl get po -n mysql -o wide # discover which k8s worker node is holding mysql-k8s primary
sshuttle -r ubuntu@<IP of k8s-worker-node-above> <pod-to-pod CIDR, e.g. [192.168.0.0/16](http://192.168.0.0/16)> -e 'ssh -i ~/.local/share/juju/ssh/juju_id_rsa'

Get the mysql password and connect directly with mysql-k8s primary pod:

juju run mysql-k8s/0 get-password

Connect to the mysql-k8s primary pod and get the count of writes:

prev=0; while true; do echo "$(date): $curr (+$((curr-prev)))";  prev=$curr; curr=$(mysql -h<POD IP> -uroot -P3306 -p<PASSWORD> -Be "select count(*) from continuous_writes_database.data" 2>/dev/null | tail -1); sleep 1; done

--- Results --- Wait for the deployment to reach a steady state, check the results of the loop above. Results with mysql-k8s / mysql-test-app related: https://pastebin.canonical.com/p/v8YVtXrX56/

Remove the relation between them and relate test-app with mysql-router:

juju remove-relation mysql-k8s mysql-test-app
# Rerun the command above, until relation is not found
juju relate mysql-router mysql-test-app

Wait until the cluster reaches a steady state, check the results of the loop above. Results with mysql-router / mysql-test-app related: https://pastebin.canonical.com/p/BPcBZKzQjR/

It can be seen that QPS has fallen from ~280 qps to ~170 qps when related to mysql-router.

github-actions[bot] commented 12 months ago

https://warthogs.atlassian.net/browse/DPE-2583