microsoft / azure-load-testing

MIT License
22 stars 2 forks source link

[BUG] Pipeline task returns InvalidPassFailCriteria when yaml has more than 5 failureCriteria #108

Open leandrodotec opened 1 year ago

leandrodotec commented 1 year ago

Describe the bug AzureLoadTest@1 - Azure Load Testing v1 task fails pipeline with error code: 'InvalidPassFailCriteria' when failureCriteria contains more than 5 items.

To Reproduce Steps to reproduce the behavior:

pipeline.yml:

 # variables are set before this, other details removed for brevity
stages:
    - stage: LoadTest
      displayName: Load test
      jobs:
          - job: Run
            displayName: Runs a fixed load test in azure
            steps:
              - task: AzureLoadTest@1
                displayName: Azure load test
                inputs:
                  azureSubscription: $(serviceConnection) 
                  loadTestResource: $(loadTestResource)    
                  resourceGroup: $(loadTestResourceGroup) 
                  loadTestConfigFile: src/HomePage/HomePage.yml   

src/HomePage/HomePage.yml:

version: v0.1
testName: BHB_Home_Page
testPlan: HomePage.jmx
description: BHB Home page
engineInstances: 1
splitAllCSVs: False
failureCriteria:
  - AppointmentType: percentage(error) > 1 
  - AppointmentType: p90(response_time_ms) > 1000 
  - PatientLocationById: percentage(error) > 1 
  - PatientLocationById: p90(response_time_ms) > 1000 
  - Provider: percentage(error) > 1
  - Provider: p90(response_time_ms) > 1000 
env:
  - name: JM_RampTime
    value: 30
  - name: JM_EndRpsA
    value: 100
  - name: JM_EndRpsB
    value: 100
  - name: JM_HoldFor
    value: 300
  - name: JM_CoolDown
    value: 10
  - name: JM_PauseFor
    value: 5

result from DevOps pipelines

Starting: Azure load test
==============================================================================
Task         : Azure Load Testing
Description  : Automate performance regression testing with Azure Load Testing
Version      : 1.2.9
Author       : Microsoft Corporation
Help         : https://docs.microsoft.com/azure/load-testing/tutorial-cicd-azure-pipelines#azure-load-testing-task
==============================================================================
{ error:
   { code: 'InvalidPassFailCriteria',
     message: 'Invalid pass fail criteria "Invalid PassFailCriteria" .',
     target: null,
     details: null } }
##[error]Error in creating test: bhb_home_page
Finishing: Azure load test

Expected behavior

Azure load testing task should allow create/update test with the same amount of test criteria as the tests created by portal.

Screenshots

Error from pipeline: image

Additional context Add any other context about the problem here.

AB#1642808

Sachid26 commented 1 year ago

@leandrodotec ... Noted. we will update once we have rectified the issue.

vinhent commented 1 year ago

We are having the same issue