[Feature] Support remote storage for visual regression testing

[gskierk]

Let us know what functionality you'd like to see in Playwright and what your use case is. Do you think others might benefit from this as well?

It comes from the comment linked below. I didn't find that issue, so I'm creating one by myself. It's about integrating visual regression testing in Playwright with popular stores (e.g., Amazon S3, Azure Blob Storage).

https://github.com/microsoft/playwright/issues/28626#issuecomment-1863431807

[pavelfeldman]

Could you elaborate on what exactly you'd like to see?

[gskierk]

Currently, the snapshots are saved on the local machine, and the use case I see is for visual regression testing in CI without the need to store the snapshots in a Git repository but rather in the cloud that we have access to (without the need to purchase external visual regression services).

[pavelfeldman]

Your request is clear, thanks!

[joeljeske]

Similar to https://github.com/microsoft/playwright/issues/28005

[akellbl4]

It would be great to see something that can help to compare screenshots with remote storage. Currently it's the only thing that we are missing and it pauses us from using Playwright.

As temporary option I was thinking of a custom matcher that would use the original one and load screenshot before it's called. I assume it can be not the fastest approach and I'd love to see a native solution from Playwright here.

[scotto-at-niche]

Ha, I was just coming on here to log that issue and I see someone's already jumped on it. Thanks @gskierk!

We're running into pretty big usability issues with our custom solution for this; and as we explore going deep on visual testing through a Playwright/Storybook integration, it would be great to have native support for cloud-based visual diff (Amazon S3 would be my team's own use case). Not being able to do this natively is hindering buy-in to Playwright from the broader dev group my SDET team supports.

[zinaelnahel89]

Same here we also would need this feature as well as a button that would accept or refuse to save the new screenshot if there are changes . Can you tell us when will this approximately be implemented ?

[JonnyPriceSC]

@pavelfeldman are there any updates on this would be really great to have the ability to store the expected snapshots remotely? The recommendation for GitHub repos is less than 5GB if we have 1000s of screenshots that are needed to test across different browsers we would really need this ability to unlock scalable regression testing as the repository advised limits will not support this currently.

[lukeapage]

Our setup is we run screenshots for a branch and then in the pipeline, make sure one is kicked off or completed for the equivalent commit on master. Then once both complete we run a screenshot comparison build. We keep the screenshots on a shared drive and have a service that keeps the image locations, hashes, state and serves a website to approve branch changes which when approved sets a pr policy to approved.. it has the great advantage of not having conflicts to merge but it’s pretty bespoke and also there is a lot of integration work with in our case azure devops.

it would be great if something was built with plugins so you could mix and match storage providers and source code providers.

[sizzle168]

Our project would benefit from such a feature as well! Are there any updates on that topic?

[kyogitq]

It would be a massive support for our project too if we get this feature within Playwright.

[sudharsan-selvaraj]

I was also looking for a similar solution and end up building one. refer to https://github.com/sudharsan-selvaraj/playshot

What it covers:

1. Support for Multiple Cloud Storage Options: Currently, the library supports uploading screenshots to Amazon S3 and SFTP servers. However, it can be easily adapted to work with other cloud storage providers.
2. Automatic Screenshot Upload: Screenshots are automatically uploaded to the remote server if there are test failures, eliminating the need to save them to version control.
3. Screenshot Updates: Users have the ability to individually update screenshots when there are changes in the UI.
4. Flexible Validation: The library provides methods for validating visual regressions for both entire pages and individual web elements.
5. Support for Soft Assertions: Built-in support for soft assertions is included, allowing the test suite to continue running even if a soft assertion fails, without interrupting the execution of subsequent test steps.

[zinaelnahel]

@sudharsan-selvaraj ohhh that's awesome . Just a quick question. We already implemented the deployment to the S3 AWS bucket and it is working fine we want only the screenshot updates part. Did you implement anything that checks the screenshot in the report and if there is a difference to highlight it and ask if we want to save the new screenshot or not ? https://github.com/microsoft/playwright/issues/30244 basically a button that approves or disapproves changes. And if you implemented this how can I use it without the part of the S3 AWS deployment because this is working already fine for us.

[sudharsan-selvaraj]

@zinaelnahel Yup, the library has the support for updating the screenshot if there is any change to the UI. You can refer https://github.com/sudharsan-selvaraj/playshot/blob/main/src/matcher.ts#L116 for the implementation details.

[zinaelnahel]

@sudharsan-selvaraj wow that's great and is there a way to use this library without this adapter part because I already have the S3 bucket part working and the screenshots as well as the report are deployed there. I just want to import the PlayShotMatcher and use it directly is that possible ? import { test as BaseTest, expect } from '@playwright/test'; import { PlayShot } from 'playwright-cloud-visuals';

const playshot = new PlayShot({ adapter: s3Adapter, // or sshAdapter for SFTP remotePathDelimiter: 'screenshots', });

const test = BaseTest.extend <{ playShot: PlayShotMatcher }> { playShot: async ({ page }, use, testInfo) => { await use(playshot.createMatcher(page, testInfo)); }, };

export default test;

[sudharsan-selvaraj]

@zinaelnahel Could you please open a new issue on my repository so that we can continue our discussion there? This will help us keep the current thread focused and organized. Thank you!

[sputnyq]

Hey @pavelfeldman. Any news on the feature :) ?

[deviantintegral]

I was originally interested in this feature because I had thought GitHub couldn’t display images in pull requests stored in GitHub LFS. That’s not the case! You get the full image comparison in PRs as well as easy browsing of the repository directly.

I’m sure there are cases where LFS won’t work. But for those who just want to commit screenshots without blowing up your repo size, consider just using LFS.

Given this is just documenting an existing feature, I think it would be a good improvement to call out or even recommend using LFS in the Playwright docs.