This custom Azure Pipeline task makes it possible to update your Azure DevOps Test Plan using the results of your test automation suite, regardless of which test framework you're using (Cucumber, JUnit, xUnit, others).
- task: PublishTestPlanResults@1
inputs:
testPlan: 'Test Plan 01'
testResultFormat: xUnit
testResultFiles: $(Build.StagingDirectory)/testResults.xml
Results from your Test Automation are available in Test Plans / Runs:
Progress is reflected in your Test Cases:
And overall progress is reflected in the Test Plan report:
It's helpful to understand the following about Azure Test Plans:
The combination of Test Case + Test Configuration is referred to as a Test Point.
When we publish our test results from our test framework to the Test Plan, we are interested in creating a Test Run that contains an outcome for each Test Point. There is a one-to-one mapping between each individual automated test and a Test Point.
The PublishTestPlanResults
task supports several different mapping strategies to help you link your test framework results to Test Cases (Test Points). The default auto
strategy will attempt to find the appropriate strategy for you, or you can customize the configuration to meet your needs.
The following strategies are used to match the test automation result to individual Test Point items:
TestName
: the name of the automated test must match the title of the Test Case.
Hyphens, spaces and underscores are removed from both the automated test name and the Test Case title to simplify case-insensitive comparisons.
Note:
This approach might be the easiest to configure, but is prone to mismatch errors if either the name of the Test Case or test method change. We recommend using the
RegEx
orTestProperty
strategies to avoid mismatch issues.
RegEx
: the id
of the Test Case appears in the name of the test automation name.
The regex to find the test case identifier can be customized by setting the testCaseRegex
.
Most test frameworks should allow you to specify the Test Case identifier somewhere in the name of the test automation.
// xunit
[Fact]
public void TestCase1234_ShouldAddTwoNumbers()
{
}
// junit
@Test
public void TestCase1234_shouldAddTwoNumbers() {
}
// cucumber
Scenario: TestCase1234 Can add two numbers
// javascript
it('Should add two numbers TestCase1234', () => {
});
// pester
it 'Should add two numbers TestCase1234' {
}
TestProperty
: the id
of the Test Case appears in the meta-data for the test.
The meta-data element to use defaults to TestCase
, but this can be customized by setting the testCaseProperty
setting.
Each test framework supports mechanisms to attach property meta-data in the output, some examples:
// xunit
[Fact]
[Trait("TestCase", "1234")]
public void ShouldAddTwoNumbers()
{
}
// nunit
[Test]
[Property("TestCase","1234")]
public void ShouldAddTwoNumbers()
{
}
// cucumber
@TestCase=1234
Scenario: Add Two Numbers
// javascript
it('Add Two Numbers @TestCase=1234', () => {})
it('Add Two Numbers #TestCase=1234', () => {})
If your Test Plan supports multiple Test Configurations, you'll need a mechanism to map the test results to the appropriate Test Configuration.
If you've specified a testConfigFilter
then only TestPoints for that Test Configuration will be updated. However, if your test-automation contains the outputs from multiple configurations, additional configuration settings are needed:
Config Aliases
: the testConfigAliases
setting is used to configure one or more simple name aliases for the Test Configuration in your Test Plan. For example, "x86_chrome" can be used in the tests instead of "Chrome on Windows 11".
ConfigProperty
: the configuration name or alias is defined in the meta-data for the test automation result. The testConfigProperty
setting is used to define one or more test-meta properties to inspect.
In this example, we're using a trait named TestId to hold the id of the TestCase in our TestPlan.
[Fact]
[Trait("TestId","12345")] // 12345 is a TestCase identifier in our TestPlan
public void AddItemToShoppingCart() {
// ... test automation
}
In our Azure Pipeline, after running your tests, you can update your Azure Test Plan Test Plan 01 using the following syntax:
- task: PublishTestPlanResults@1
inputs:
testPlan: 'Test Plan 01'
testResultFormat: xUnit
testResultFiles: $(Build.StagingDirectory)/testResults.xml
testCaseMatchingStrategy: property
testCaseProperty: TestId