edumserrano
commented
2 months ago I understand this is just a quality of life improvement, there's almost nothing that this feature allows doing that you can't already do. However, let me give you an example of how my Azure DevOps CI pipeline looks like to help justify the use case for my feature request:
[!NOTE]
The below are fragments of my pipeline for Azure DevOps that uses the blob reporter and the merge cli to produce the merged Playwright html reporter.
First, each job will run its sharded Playwright test run and upload the raw data from the blob reporter. The $(shardValue) and $(totalShards) are variables set for each shard run. After each sharded Playwright test execution, the pipeline will contain several artifacts named playwright-blob-reporter-shard-<shard-number>, each containing a zip of the raw data from the test execution.
- task: PowerShell@2
displayName: Run Playwright tests
inputs:
targetType: inline
script: |
npx playwright test --shard=$(shardValue)/$(totalShards)
- task: PublishPipelineArtifact@1
displayName: Publish Playwright blob reporter
inputs:
targetPath: $(Build.SourcesDirectory)/test-reporters/playwright
artifactName: playwright-blob-reporter-shard-$(shardValue)
parallel: trueAfter all the sharded runs have executed, we download each of the artifacts and flatten them into the ./blob-reporter directory. At this point, the ./blob-reporter directory contains all the zip files from each sharded run and all we have to do is use the Playwright merge CLI and give it a Playwright merge config to produce the merged test reporters.
- task: DownloadPipelineArtifact@2
inputs:
buildType: current
itemPattern: '**/*.zip'
targetPath: $(Build.ArtifactStagingDirectory)/temp
- task: PowerShell@2
displayName: Merge playwright blob reports
inputs:
targetType: inline
script: |
mkdir $(Build.SourcesDirectory)/blob-reporter
Get-ChildItem -File -Recurse -LiteralPath $(Build.ArtifactStagingDirectory)/temp `
| where {$_.FullName -like '*blob-reporter*'} `
| Move-Item -Destination $(Build.SourcesDirectory)/blob-reporter
npx playwright merge-reports --config=playwright.merge.ts ./blob-reporterTrying to do the same with monocart reporter is definitily possible but involves more steps as well as it involves keeping the merge.ts file in sync with how many shards we want to use. This is not an issue with the Playwright merge CLI example I just gave because we can provide it with a directory of zip files and it will read all of them to generate the merged test reports.
cenfun
Description
After #139, with version
2.7.1of monocart-reporter it became much easier to merge reports from sharded Playwright test runs.I can easily create a merged monocart report with the following merge file:
And by running:
npx tsx .\playwright.monocart-reporter-merge.ts.In a CI environment, to get the above to work, I have to: 1) upload the monocart reporter output generated from each sharded run to the pipeline artifacts. 2) once all shards have executed, download the artifact from each sharded run into a folder structure that matches the one expected by the merge file above. 3) run the merge command:
npx tsx .\playwright.monocart-reporter-merge.ts.Whilst it is possible to do all of the above it is a bit cumbersome in two aspects:
1) If you want to change the number of shards or the directory where the data from the shards is you have to update the merge file logic as well. 2) You have to code some logic on the pipeline so that the download from the pipeline artifacts matches the folder structure expected by the merge file.
I believe that both of these issues can be alleviated by following the approach used by Playwright with the blob reporter and the merge cli.
Suggestion
1) Consider adding a way to tell monocart reporter that it should only produce raw data and that at the end that raw data should be zipped. For instance, the
MonocartReporterOptionscould have arawboolean property which defaults tofalse. 2) Create a CLI tool, whether as part of the existingmonocart-reporteror as a separate package, that can be invoked to merge monocart reports. This CLI tool would take in a path to a folder that contains one or more zipped raw report data and invoke themergefunction from themonocart-reporter. In essence, this CLI tool is nothing more than a wrapper around themergefunction from themonocart-reporterpackage. 3) The merge CLI tool should also take in an input parameter that would let us specify the path to a merge file. This merge file should export theMonocartReporterOptionsobject which would be used to do the merge.