MYSQL and Redis Hitting 80-100% Utilization

[nelsontalag]

Good day all,

Our Moodle setup has been running well in the past few days at about 2000 users concurrent connections.

the last 3 days there are about 80k new users registered

but now about 20k concurrent users are now trying to use the system and we are now hitting max utilizations

we have scaled up DB and Redis and running 10 vms in the vmsss

turned on database persistent connections in moodle config also. Note that prior to the persistent config change we are also hitting max utils.

any tips on how we can reduce and fine tune system performance ?

thanks all

[UmakanthOS]

Flagging with @naioja and @savjani

[savjani]

@nelsontalag - Can you please open a support ticket from Azure portal

[naioja]

@nelsontalag try to find out what is causing the bottleneck

• look at your VM SKU used in the VMSS if it supports "Advance Networking" https://docs.microsoft.com/en-us/azure/virtual-network/create-vm-accelerated-networking-cli#supported-vm-instances
• check how the total network bandwidth is used : either by customer traffic, traffic to the OS disk or to other PaaS services like you mentioned Mysql and Redis

If any of those are red lined switch to a higher VM SKU : https://docs.microsoft.com/en-us/azure/virtual-machines/dv2-dsv2-series#dsv2-series Moreover look at the attached OS disk and check if you are hitting an IOPs limit and try to increase the size of the disks : https://docs.microsoft.com/en-us/azure/virtual-machines/disks-types#disk-size-1 , the 1TB or P30 i found out in my tests that they deliver linear performances.

On the Mysql DB side look at the obvious metrics like CPU and IOPS and increase accordingly, alternatively I've used the new PaaS offering called "Flexible Server" : https://docs.microsoft.com/en-us/azure/mysql/flexible-server/concepts-compute-storage where you can tune for an exact amount of Cores and IOPs

On the LoadBalancer side look at the Use SNAT Ports and if you are hitting that number add more Public IPs for outbound traffic (if you have our default installation those need to be added to the Mysql Firewall in Azure)

Finally not everything needs to be cached in Azure Redis meaning that you can setup another decently sized VM using memcache and point some of your Moodle caches to use that one.

[UmakanthOS]

@nelsontalag Please re-open with any extra context. Closing per prior guidance.