search

microsoft / playwright

Playwright is a framework for Web Testing and Automation. It allows testing Chromium, Firefox and WebKit with a single API.
https://playwright.dev
Apache License 2.0
66.35k stars 3.63k forks source link

[Feature]: Label screenshots made in playwright with test file name #33143

Open mastrzyz opened 2 days ago

mastrzyz commented 2 days ago

🚀 Feature Request

Append the sanitized test name to the automatic screenshot mechanism of Playwright.

Why?

Currently the names are very generic and cause conflicts in CI/CD Systems like Azure Devops when we have multiple jobs failing with the same generic name:

Image

It's also not very clear which number responds to which testcase when we are able to upload:

Image

Example

Fix!

Added the sanitized file name, a good example of how this looks like are the fixed unit tests :

Image

This will allow us to find the artifact easier and not conflict in CI/CD systems like in Azure Devops

Motivation

This is purely a CI/CD Improvement for Playwright integration for us, we have a massively parallel system running hundreds of tests across multiple shards

dgozman commented 1 day ago

@mastrzyz Thank you for filing the issue!

Don't you have the same problem with traces, all named trace.zip? It seems like the better solution is to update your reporter to include the test name in any attachments it generates, since these are presented as a plain list, not grouped by test.

mastrzyz commented 1 day ago

Sorry @dgozman , I'm not using any custom reporter and relying on the existing Playwright reporters which are outputting trace.zip and screenshot1.png.

dgozman commented 15 hours ago

@mastrzyz In this case, why do you only focus on the screenshots? I'd think that trace.zip would be equally important. Do I miss something?

If I understand correctly, you are using junit reporter that is uploaded to Azure DevOps through this task, don't you? Perhaps we can improve that part instead? Why does it not show the full attachment path? Reading this post, I would think the full attachment path that already includes test path could be shown. Not to mentioning that it's a part of the specific test output.

mastrzyz commented 5 hours ago

You know to tell you the truth, why don't I see this failure for trace.zip!

Image

Could it be that trace.zip is only one? we have retries enabled so I wonder if we take screenshots in first attempt and then seccond also takes and collides