percy / percy-agent

[Deprecated in favor of `@percy/cli`] An agent process for integrating with Percy.
https://percy.io
MIT License
22 stars 25 forks source link

Snapshot comparison issues #603

Closed ChaitraVishwanathTR closed 3 years ago

ChaitraVishwanathTR commented 3 years ago

Snapshots are not compared with baselined snapshots for consecutive runs. Instead compares with previous run snapshots. a. Baseline Build is approved. Next run showed differences. Marked 'Requested for Change'. Issue is fixed . Next run the Site Snapshot is showed as difference(unreviewed) though after fix it matches with baseline snapshot.

b. Baseline Build is approved. Next run showed differences(unreviewed ) snapshots. Did not take any action. In next run , new snapshots matched the previous run (not the baseline snapshot) and are marked as unchanged. Though they differ from baseline snapshot

c. There is no baseline build. And multiple runs given (eg 3). Each build showed all snapshots as unreviewed. Latest build shows entire site in Red color (right side snapshot)

d. Sometimes if there is a change in commit number, all snapshots are showed as unreviewed. But sometimes though the commit number changed , it shows only difference snapshots as unreviewed .

We are executing tests from Jenkins and selenium/percy code is in Github.

Robdel12 commented 3 years ago

Hey @ChaitraVishwanathTR thanks for the issue, unfortunately this isn't an SDK issue (so I'll be closing the issue). You can reach out to support here: https://www.browserstack.com/contact#technical-support

It would also be worth reviewing the baseline picking docs: https://docs.percy.io/docs/baseline-picking-logic One thing to call out from there is approvals have zero impact on baselines. It's all facilitated via git.