Create POC in Percy

[pjhill]

Description

Using a free account set up a Percy scan of the va.gov landing page. Attempt to tease out any unknown unknowns related to using this tool for production monitoring.

Things we do not need to do

• Don't need to connect this to the PagerDuty service at this time
• Don't need to create to create a Canary with screen shots in the script

[holdenhinkle]

For my reference - AWS Synthetic Canary

I now have permissions to create a Canary - https://dsva.slack.com/archives/CBU0KDSB1/p1625067149442700

[holdenhinkle]

Draft PR - https://github.com/department-of-veterans-affairs/devops/pull/9508

I added the Percy token to AWS Parameter Store.

I'm almost done with the Node.js Canary script. I should have that finished early tomorrow morning. Then I have to figure out how to publish it, with the node_modules directory, following the instructions here - https://github.com/department-of-veterans-affairs/devops/blob/master/scripts/aws-canary-scripts/readme.md

I'm pretty close to completing this ticket.

[holdenhinkle]

Links for my reference:

terraform-aws-vsp-canary README - https://github.com/department-of-veterans-affairs/terraform-aws-vsp-synthetic-canary/blob/main/README.md

Writing a Node.js canary script - https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch_Synthetics_Canaries_WritingCanary_Nodejs.html

AWS SDK, Class: AWS.SSM - https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/SSM.html#getParameter-property

[holdenhinkle]

New temp repo for the percy canary script - https://github.com/department-of-veterans-affairs/percy-canary

[holdenhinkle]

The folks at Percy where super responsive today.

I opened an issue with them and they responded within in minutes - https://github.com/percy/cli/issues/403

I also emailed them because I needed a read request token and, again, they responded within minutes:

Hi Holden, Thank you for your reply. For your query, you are supposed to export the token generated from https://percy.io/api/v1/tokens to PERCY_TOKEN The npm package reads these values as process.env.PERCY_TOKEN. Hope it helps.

I've got the integration with Percy working, with some unexpected behavior from Percy's workflow. I demoed it to Peter, and Percy's workflow isn't what he was expecting either. I'm going to reach out to Percy about this.

Tomorrow I'll throw this into a Canary. I'll also start on the Applitools POC.

[holdenhinkle]

I sent the following email to Percy support:

---

Hi Harsh,

Thanks, I got everything working.

I have a question about Percy's workflow, which is not what I was expecting.

First of all, I'm using the following Percy modules in an AWS CloudWatch Canary to monitor the va.gov homepage for visual regression: "@percy/client": "^1.0.0-beta.58" "@percy/core": "^1.0.0-beta.58"

We are NOT using Percy in CI. Also, the AWS Canary isn't running yet.

Here's Percy's workflow:

1. Take snapshot that matches the approved snapshot: => auto-approved

2. Take snapshot that DOES NOT match the approved snapshot: => requested change, unreviewed snapshot

3. Ignore request (the 'flagged' snapshot is not rejected nor approved)

4. Take snapshot that DOES NOT match the approved snapshot, but matches the last not-approved snapshot: => auto-approved

I would have expected step 4 in my example to NOT be auto-approved, because the snapshot was not reviewed. I expected that Percy would again flag it as 'requested change, unreviewed snapshot'.

Is it possible to configure Percy to work as I expected it to? That is, to 'flag' snapshots that don't match the last approved snapshot?

Thank you very much,

Holden

[Robdel12]

👋🏼 I can hop in to answer really quick while my dinner finishes up 😆 Percy's base picking logic is facilitated by git (and it doesn't take approves/rejects into account). Here's the docs on how the base picking logic works: https://docs.percy.io/docs/baseline-picking-logic

Auto-approvals also happen if they're on a matching branch -- this is mainly for the mainline dev branch. Whenever you merge feature --> main, there will always be changes, and those changes are assumed to be approved in the feature branch workflow before merging into the main dev branch (so no need to reappove those same changes once they are merged). You can go into the project settings to add/remove branch names from auto-approval.

With that said, approvals have no impact for how baselines are picked. It sounds like you're currently running all the builds on master (taking a guess here :p)? If so, that will set the baseline on each build (and by default that branch is also set to auto-approve). Builds on the baseline branch will be used as baselines for other branches. So you can branch off of master to a feature branch and Percy will look up the common commit between the baseline branch & the current branch (to show only the changes made in the current build).

Hopefully that helps clear things up!

[holdenhinkle]

Thank you, Robert! Your explanation helped a lot. I really appreciate the support you gave us yesterday. Happy Friday to you!

[holdenhinkle]

This is done.

[holdenhinkle]

Hey @Robdel12 I thought I'd circle back to this quickly to confirm my findings with you, just to make sure I didn't miss anything.

We're interested in using Percy outside of CI (outside of GitHub) to run a visual regression test against the va.gov homepage in an AWS CloudWatch Canary, which will run every 5 minutes.

If a difference is detected in a Canary run and the build in Percy isn't reviewed, we want the same difference to be detected (if the difference persists, of course) in subsequent Canary runs until someone manually reviews it.

Is it possible to do this with Percy? It seems like it isn't, so I wanted to double check with you.

I've tested Percy with our Cypress tests running in parallel in Jenkins and it works great. We're hoping Percy can handle this Canary use-case too.

Thank you!

[holdenhinkle]

from:	Percy Support support@percy.io
reply-to:	Percy Support support@percy.io
to:	holden.hinkle@oddball.io
date:	Jul 12, 2021, 8:48 AM
subject:	Re: 462318-Percy outside of CI? Outside of GitHub?

Hi Holden, ​ Thank you for reaching out to us. ​ The Github SCM integration is specifically used to prevent merges when the build against a PR had any Diffs. Upon approval, the merge option is enabled.

The Approval workflow specifically works for the SCM. ​ In case you configure the project outside Github, the approvals will not have any effect. The diffs will visible on the dashboard. ​ Also, when running a test, Percy does not fail the CI build in case a diff is encountered. This is because Percy diffing is async to the CI run.

Let me know if you have any further questions. ​ Regards, Shawn BrowserStack Support

[pjhill]

POC complete -- we can review the results in the future if required.