Open felipmiguel opened 1 year ago
Hi @felipmiguel, Thank you for submitting this issue. We will try to reproduce this issue and figure out how to fix it.
Hi @felipmiguel,
I have reproduced this issue. The RPS drops during application scale-out, see metrics from Application Insights. From dashboard on test client(Azure Load Test) side, response time increased during app scale-out. It can also be observed from server side, see metrics from Application Insights
From Application Insights live metrics, we can see part of requests are handled by new Pods which have longer response time during new Pods warm-up. The application is using in-memory cache, the response will be longer in newly created Pods than existing Pods because the cache is not available yet.
According to RPS calculation formula: Virtual users = RPS * latency in seconds. Given the number of virtual users is consistent, as latency increases the RPS drops. This explains why the RPS drops during application scale-out, it is expected behavior. See more details here: Key concepts for Azure Load Testing | Microsoft Learn
Describe the bug During scale out operations the performace of the application is degraded. It happens with auto-scale and also manual scale. It doesn't happen during scale-in operations.
To Reproduce Steps to reproduce the behavior:
Expected behavior The expected behavior is that the performance of the application stays stable during the scale-out operation. Once the operation is complete, then throughput should improve. That is specially important for systems under heavy load. If there is auto-scale in place usally is triggered when more performance is needed, but during this scale-out operation process the performance of the solution goes down.
Screenshots
Additional context To generate the load I was using Azure Load Testing and the load was constant, that is why it is possible to detect this situation.
Can we contact you for additional details? Y/N
If yes, please send us your contact information to AzureSpringCloud-Talk@service.microsoft.com and include the issue number in the email title.