search

alexneo2003 / playwright-azure-reporter

Playwright Azure DevOps Reporter
MIT License
67 stars 11 forks source link

[Bug] Azure DevOps pipeline always succeed when missing permissions to publish test results. #64

Closed AndrewRiggs-Atkins closed 5 months ago

AndrewRiggs-Atkins commented 5 months ago

When running Playwright tests with the reporter in an Azure DevOps pipeline, if it does not have permissions to publish the test results the job/pipeline always succeeds. This is irrespective of whether the tests pass or fail.

This causes developers to view the pipeline as succeeding, despite the Playwright tests themselves failing.

The expected behavior would be that if the reporter is unable to successfully publish the test results, the job/pipeline fails. This is preferable to having failing tests display as passing in the pipeline (and unknowingly being pushed to production).

If the permissions are not set, the following error is raised, and the pipeline reports a success.

  1 failed
    [XXX] › test-cases/XXX ... 
  4 passed (5.9m)
azure: TF237026: You do not have  the appropriate permissions to publish test results in this team project.
azure: Failed to create test run. Reporting is disabled.
alexneo2003 commented 5 months ago

@AndrewRiggs-Atkins as I understand you correctly you want fail pipeline when tests failed? If it's true - it's not problem of this reporting, you need correct configure your pipeline tasks. Each one ADO tasks has own parameter continueOnError. If you set it to false it should interrupt pipeline when task failed. Also you can read about task parameter Run this task

Anyway, what you want is not the reporter's responsibility. Configure the pipeline correctly

AndrewRiggs-Atkins commented 5 months ago

@AndrewRiggs-Atkins as I understand you correctly you want fail pipeline when tests failed? If it's true - it's not problem of this reporting, you need correct configure your pipeline tasks. Each one ADO tasks has own parameter continueOnError. If you set it to false it should interrupt pipeline when task failed. Also you can read about task parameter Run this task

Anyway, what you want is not the reporter's responsibility. Configure the pipeline correctly

By default continueOnError is set to false, and this is the case in my pipeline (ie. it is not defined). As such the expectation is that when the test fail, the task fails.

This is true when the permissions are set to publish test results, but when they are not the pipeline does not fail as expected and instead reports success for the Playwright tests (even though they failed).

alexneo2003 commented 5 months ago

Test Run with tests results and Release Pipeline not related in any way. They can exist independently of each other. Release Pipeline with tests should be failed (or not) regardless of that there is Test Run or not and what result it has Reporter can't interrupt or fail you own pipeline. Reporter can only publish test results or/and log results inside pipeline logs. Playwright can fail task with tests and it should do this