Investigate Test Failures on Azure DevOps Pipeline Instance

[vbamagician]

Problem Overview

I have developed an automated testing script for the CPQ web application using C# .NET, Playwright, and SpecFlow. The script was functioning as expected in my local environment. However, after deploying the repository with all the commits, the tests failed to execute successfully on the Azure DevOps Pipeline instance.

Issue Details

The tests run smoothly on my local machine, but there seems to be an inconsistency when executed in the Azure DevOps Pipeline. Unfortunately, the lack of visibility into the pipeline runtime environment makes it challenging to identify the root cause of the failure.

Proposed Solution

In order to diagnose the issue and pinpoint the cause of failure during the Azure DevOps Pipeline run, I propose recording a video of the test execution using Playwright's built-in capabilities. By publishing this video as an artifact, we can analyze the test run behavior and identify any discrepancies between the local environment and the Azure DevOps Pipeline instance.

Action Items

1. Implement Playwright Video Recording: Modify the testing script to include Playwright's video recording functionality during test execution.

IPlaywright playwright = await Playwright.CreateAsync();

//Browser
IBrowser browser = await playwright.Firefox.LaunchAsync(new BrowserTypeLaunchOptions
{
    Headless = false
});

_browserContext = await browser.NewContextAsync(new()
{
    RecordVideoDir = "videos/"
});

//Page
return await _browserContext.NewPageAsync();

2. Publish Video as Artifact: Configure the pipeline to publish the recorded video as an artifact after the test run.

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(System.DefaultWorkingDirectory)/Sources/CPQUIPROD/bin/Debug/net6.0/videos'
    artifact: 'videos'
    publishLocation: 'pipeline'
  condition: always()

3. Review and Analyze: Access the published video artifact and analyze it to identify any issues or discrepancies during the test execution on the Azure DevOps Pipeline instance.

Additional Information

• Local Environment: Windows, Visual Studio, Playwright, SpecFlow

• Azure DevOps Pipeline Configuration: Note that the targetPath in the YAML code is crucial, and it should be adjusted based on your project structure and the location where the videos are recorded.

• Azure DevOps Pipeline Configuration: Note that the condition: always() is added to ensure that the artifact is published regardless of the pipeline run outcome.

I appreciate your assistance in resolving this issue promptly. If you require any additional information or assistance, please don't hesitate to ask.

Thank you.

[vbamagician]

Issue Resolution

I am pleased to inform you that the issue regarding test failures on the Azure DevOps Pipeline instance has been successfully resolved. The proposed solution, which involved implementing Playwright video recording during test execution and publishing the video as an artifact, proved effective in diagnosing and addressing the problem.

Resolution Details

As the proposed solution has proven effective and the test failures have been addressed, I propose closing this GitHub issue. If you have any further questions or concerns, feel free to reach out.