Azure Load TestinLoad Tests Return 'Not Applicable' Failed status despite showing as successful in Exported Result Files & AppInsights[BUG]

[MonikaReddy-MSFT]

Customer has raised an issue in Microsoft Q&A - https://docs.microsoft.com/en-us/answers/questions/684415/index.html

We are currently running a POC on the new Azure Load Testing feature (currently in preview) as a resource to run our jMeter tests against an in development API.

Unfortunately we are hitting a few problems with the output results displayed within a test run, namely the tests always fail with no start, end and duration time along with a 'Not Applicable' status when running against the APIs despite all tests running successfully directly within the jMeter GUI. The same tests when run in the Azure Load Testing tool also appear to show as 'Passed' when exporting results from the test screen.

Has anyone experienced this before and would be able to help, as this tool could be exactly what we are looking for if we can get it to output the correct results.

Our jMeter test is fairly simple and obtains a validation token which is used in the latter request, I have manually verified this is all authenticating correctly and the variables we are using in these tests are being correctly populated when run in Azure. Note this issue appears in both our GraphQL and REST API tests.

See attached screenshots:

Test Result wD6KP.png

Exported Result aTjYQ.png

We have also attempted to use the 'Ultimate Thread Group' plugin following a suggestion on StackOverflow, but unfortunately this also appears to return the same results (passes when run locally or through Blazemeter) but returns no result in Azure Load Testing despite the requests returning a successful result in AppInsights.

If anyone would be able to help at all this would be greatly appreciated. Unfortunately at the moment this is preventing us from reccomending Azure as a tool for our upcoming Load Testing strategy .

AB#1461304

[ChrisTyler-6788]

Hi, Thank you for raising this issue on my behalf.

Please find additional notes I have provided on the Microsoft forum:

Since raising this issue we have since carried out some additional investigations on our side and noticed that there appears to be a particular issue when using the GraphQL HTTP Request type that was added to jMeter in version 5.4. When using this type of request for a valid graphQL query. The request is processed successfully locally within jMeter and also within Blazemeter, but not within the Azure Load Testing tool (which provides the issues displayed above).

I have since been able to get a valid response from the Azure Load Tester, when I submit my GraphQL requests directly within the body of a standard HTTP Request and with the 'Ultimate Thread Group' plugin (without this the response fails). Would it be possible to confirm if the GraphQL HTTP Request type from JMeter is currently supported for the Azure Load Testing tool, or if this is planned as part of a future update?

[abranj1219]

Hello @ChrisTyler-6788 Thanks for reporting this. We are looking into this issue. As you reported it seems the tests have executed correctly,there might be some execution errors. Would you be able to share some more details like testrunName, jmx to speed this up.

[krishna1s]

Hi @ChrisTyler-6788 can you also share the region name in which you have created the load test along with working and non working JMX.

[ChrisTyler-6788]

Hi @abranj1219, @krishna1s.

Please find an example of the GraphQL requests currently failing. There are also two CSV files which contain paths to authenticate and access the API, I am unfortunately unable to provide these as this is an in development API, but this hopefully should visualize the structure of our JMeter tests.

The region we have been using is 'North Europe'

Many thanks

GitHubExample.zip

[MonikaReddy-MSFT]

@ChrisTyler-6788 -Can you try to run the test again, as this test seems to be a one-month-old test. We are hoping we might have fixed some issue in past few weeks related to this.

Please let us know once you run the test again.

[ChrisTyler-6788]

hi @MonikaReddy-MSFT.

Thanks for your reply. I have now re-run our tests and these do look to be passing now and displaying full test results.

Many thanks for your help

[MonikaReddy-MSFT]

@ChrisTyler-6788 - Glad that the issue is resolved now. We will now proceed to close this thread. If there are further questions regarding this matter, please tag me in your reply. We will gladly continue the discussion and we will reopen the issue.