search

microsoft / azure-load-testing

MIT License
22 stars 2 forks source link

how can I get Maximum number of request per seconds #54

Closed Ethan033100 closed 2 years ago

Ethan033100 commented 2 years ago

I need to generate more than 20000 requests per seconds. I created a dummy example with APIM and azure function. I tried using the same JMeter script using 250 threads for 60 seconds. I tried with 10 and 20 instances and I noticed that I was not able to achieve more than 2000 transactions per seconds. The most performant was the cluster with 10 instance, rather than the one with 20.

AB#1487102

Ethan033100 commented 2 years ago

I have a problem. My APIM is in southeastasia and azure load testing is in Australia.

I can see that the average number of users were 4264 and the p90 of the response time is 3.88 seconds.

I found an SSLException error, how can I solve it?

Ethan033100 commented 2 years ago

any updates?

kaumishra commented 2 years ago

Hi Ethan, thanks for sharing. I am adding @abranj1219 from our dev team to help with this.

Ethan033100 commented 2 years ago

@abranj1219 Love, if you could help me tweak my configuration asap.

abranj1219 commented 2 years ago

@Ethan033100 This seems to be an interesting scenario. We shall support you to generate that load. MALT can easily generate 20k rps. It would be great if you could share your APIM configuration. I believe you are already taking care of setting correct configuration on that part. But would be great to double check that and upgrading tier https://docs.microsoft.com/en-us/azure/api-management/upgrade-and-scale . Please check and revert.

Ethan033100 commented 2 years ago

JmeterConfig APIMperformance.txt these files may related. Thanks, if you can feel free to help check. @abranj1219

Ethan033100 commented 2 years ago

any updates?

abranj1219 commented 2 years ago

@Ethan033100 which tier of APIM and function app you are using, Could you try upgrading plans to premium and test. I suspect the limitation could due to the APIM and function app plans. As per this API Management pricing | Microsoft Azure, there are limiting RPS for each tier. Please get back in case you need further help

Ethan033100 commented 2 years ago

I am using APIM Premium SKU

Ethan033100 commented 2 years ago

@abranj1219

Ethan033100 commented 2 years ago

Can I get any updates on this? Or other potential cause?

harshan-b commented 2 years ago

@Ethan033100 It seems like the bottleneck is caused by APIM. Can you please check if there are any issues in Availability and Performance of APIM's Diagnose and Solve Problems tab? High round trip time in APIM can cause network timeouts (and SSL exceptions). It would be helpful if you could share more details about APIM and Azure Functions backend. Otherwise, I would suggest forwarding this issue to APIM.

EDIT: We ran a few tests with premium Functions without APIM, and both Load Test and Functions app scale as expected >15000 RPS. It seems like the issue is APIM. Please forward to APIM.

We recommend running the Load Test for longer duration (ramp up time >10mins) to allow APIM and Functions app to scale out appropriately.

abranj1219 commented 2 years ago

@Ethan033100, As @harshan-b mentioned, We are able to achieve high scale if we remove APIM layer and directly load test premium function app. Could you check if you are able to achieve desired scale without APIM. This would confirm that it is related to APIM configurations and ALT can generate that much traffic indeed,

abranj1219 commented 2 years ago

@Ethan033100 , Could you confirm this if this was indeed related to APIM configuration? Would close this issue now. Please feel free to reopen in case you need further support