tnolet
commented
3 years ago Hi Geoff, thanks for reporting and making a clear case. Indeed, screenshot diffing done well is almost a whole new company (or at least team). This does not mean we will never do it, but not in the short term.
As for the Percy SDK, importing the NPM package would be easy for us, but running our Puppeteer (or Playwright) process wrapped by the Percy agent is harder. We do a whole bunch of things to make run these processes, control them, start them, stop them, gather metrics etc. The percy exec -- syntax would make that non trivial.
💡 For general support requests and bug reports, please go to checklyhq.com/support
Is your feature request related to a problem? Please describe. We currently leverage a patched-together solution of Cypress, Percy, and Github Actions to do verification that certain important pages and workflows are running as expected after deploys. One of the most common things we spot with these tests is a subtle UI breakage or styling problem that's only visible in production either due to production data or things like production-environment CSP settings, etc. We spot these problems with Percy snapshots taken from Cypress.
We'd love to be able to leverage Checkly's checks to accomplish something similar so that we're no longer running our own solution.
Describe the solution you'd like
There are two solutions that would enable us to migrate all our existing checks over to Checkly.
1) If Checkly could alert on screenshot diffs like Percy does, we'd leverage that. We get an alert if any screenshot differs by 1px. Percy has a whole team working on this and all the edge cases around that, so that's probably a big ask!
2) If Checkly could install and allow use of the Percy agent in the test containers, we could take snapshots from within Checkly tests and continue to leverage Percy's diffing capability and notifications for changes. That seems like the best of both worlds.
Percy's agent and API leverage Puppeteer, so that would hopefully limit the number of new things that would be needed in the image.
https://docs.percy.io/docs/puppeteer
Describe alternatives you've considered
Without this we're probably going to leverage Checkly only for API calls and continue to do synthetic monitoring with our own home-built solution. That would be a bummer, because we would get big value out of Checkly features like checking from two data centers, checking for double-failures, direct triggering from Github deployments, etc.
Additional context
Since it's the question I always ask when someone wants a new feature: We'd possibly jump from Starter to Enterprise level Checkly use if we were able to move our existing suite of workflow checks into Checkly.