Skipped tests are marked as with the passed status in the testplan

[AnnEnervalis]

Description

I have a number of skipped tests that don't run yet. You can see in the tests tab that they are effectivaly not run:

However, in the testplan the tests are marked as passed:

Can you mark them as 'NotExecuted' please?

Environment Details

provide details about your pipeline environment

• build-agent: self-hosted
• test-format: webdriverIO with mocha
• test-lanugage: Typescript
• pipeline-format: yaml

If you need more information, don't hesitate to ask. (Thanks for this plugin, it really helps)

[bryanbcook]

Thanks for raising this.

To help clarify if this is an issue converting your mocha test results into a common format, or a mismatch when publishing to the Test Plan:

• Can you provide a sample of your test results?
• Were these tests previously executed and marked as Passed and the status is not being reset?

[AnnEnervalis]

These are new tests or tests that were marked previously as passed. Both have the same behavior. I can see that every run adds a 'Passed' status, so I don't think the previous state is related.

In attachment you can find the junit file. Does that help? junit-results-file.zip

[bryanbcook]

Ok. I think the issue might be a bug in the test-parser library that I'm using to convert test-formats into a common format. Your skipped tests in the JUnit xml results appear as:

<testcase ....>
   <skipped/>
</testcase>

I think the test-parser library is expecting an attribute instead of an element. I'll do some research to determine if this is a variant of the JUnit format (there are lots of variants) and log a bug for this.

[bryanbcook]

Logged https://github.com/test-results-reporter/parser/issues/70

[bryanbcook]

@AnnEnervalis just an update - the above defect has been fixed, now we're just waiting on a new release of the package.

[bryanbcook]

@AnnEnervalis new version published. I'll leave this open and hope to hear from you soon.

[AnnEnervalis]

Hi, I am afraid that it doesn't work any better. I notice that failed tests also are reported as passed. Skipped tests are also still 'passed' In attachement you can find a new junit file. If you look for testcase 146035, you see that it failed, but It marks as 'passed'

junit-results-file (2).zip

[bryanbcook]

That's frustrating! Let me see if I can repurpose your test results to one of my Test Plans and see if I can reproduce the issue. There was a bug in the test-parser library for Skipped tests, hopefully there isn't one for errors too. But looking at my code, the error could be on my end too.

I notice in your test results that 146035 appears more than once, so that might also be contributing to some of the confusion. One of the results is Skipped, another is Error.

[bryanbcook]

can you provide the YAML you're using for my extension?

[AnnEnervalis]

- task: PublishTestPlanResults@0
  displayName: 'Publish HMI Tool Results to Test Plan'
  inputs:
    testResultFormat: 'jUnit'
    testResultFiles: 'junitResults/junit-results-file-hmi-testtool.xml'
  condition: eq(variables.TESTHMITOOL, 'true')

The fact that you see it 2 times is because I rerun failed tests.

[bryanbcook]

I mention the duplicates because the matching strategy will match the first test result and discard the second. I can probably add some additional logging to reflect this.

Your test framework automatically retries tests and logs them in the result file more than once?

[bryanbcook]

I'm having some challenges reproducing your issue.

I took your jUnit xml and reduced it to 5 test cases: test-cleansed.zip:

• I modified the test numbers in your test names to match test case numbers from my test plan.
• I included duplicates to demonstrate how these are processed.
• I also reviewed the code block mentioned above, and verified through Postman that the DevOps REST API will accept string or enum values, so the extension isn't sending an invalid value.

Test Case	Test Name	Test Result	Test Case Result
167	167 Verify the Summary page error elements when all of the previous modules was unsuccessful	Passed	Passed
174	174 Verify the Summary page success elements when all previous modules were successful (skipped)	<skipped/>	Not Executed
174	174 Verify the Summary page success elements when all previous modules were successful (failed)	<failure/>	N/A - duplicate
175	175 Verify the Summary page error elements when a random previous module was unsuccessful	<skipped/>	Not Executed
176	176 Verify the Summary page error elements when a random previous module was unsuccessful	<failure/>	Failed

I published this test file using the current version of the extension and my test plan shows the results being reflected with the above. I also repeated my experiments by resetting the test cases to "Active" to ensure that they were be modified by the extension.

The progress report reflects the changes as well.

I'll release a version shortly with some additional debug logging to help you troubleshoot, but I would like to confirm that the Skipped tests are being reported as Not Executed and that we're not trying to solve multiple issues simultaneously. To do this, can you run a test suite with only a single skipped test and publish that against the test plan to rule out issues with duplicates or other interfering symptoms?

[bryanbcook]

@AnnEnervalis the initial defect of junit skipped tests was resolved and released in 0.1.14

Marking this as closed as I was not able to reproduce your issue after the fix.

If you continue to experience issues, please comment here or open a new defect.