Difference in screenshots: Github Actions run vs. Local run

[twokul]

Hello! 👋

I came across the difference in screenshots which I cannot explain. Maybe someone could land a hand or could help debug the issue?

I have created a fake change to trigger the build (change in the README file, does not impact the functionality, at all.)

When percy exec runs on Github Actions, it shows that the newly taken screenshots are different from the baseline. Build Link.

When I run percy exec locally, there is no difference. Build Link.

I have compared the environment, head & base commit and browser version - they are all the same.

In both cases, the same command is used yarn percy exec yarn test:visual; yarn test:visual is a shortcut for ember test --filter='visual acceptance'.

The only thing that is different is the $PERCY_BRANCH variable which I do not set in Github Actions. If I read the documentation correctly, I do not need to set it, it defaults to the current git branch.

[mixonic]

I believe the variable is set at https://github.com/percy/exec-action/blob/be2fc13289392ce70a89314173ae2a5632271189/src/index.js#L10 ?

[Robdel12]

Hey @twokul! This doesn't appear to have anything to do with the env vars or the action. FWIW, if you're using @percy/cli the GH action isn't needed anymore. You can see an example of how to move away from the action here

Looking at the two builds, it looks like the inline attrs on the SVGs for the graph is changing.

vs

Notice the width attr changes from width="89.31522369384766" to width="79.57142639160156"

This could be "normal" (not known until now) flexing of the graphs between test runs or it could be a test browser width/viewport difference between your local & CI. It appears the graphs are responsive so they'll change according to the browsers width, so that would have to be exactly the same across test runs

[twokul]

@Robdel12 thanks for the response! A few follow up questions.

Looking at the two builds, it looks like the inline attrs on the SVGs for the graph is changing.

I wish I could do this sort of debugging locally 😄

This could be "normal" (not known until now) flexing of the graphs between test runs

How does one go about stabilising the screenshots in this case?

it could be a test browser width/viewport difference between your local & CI.

This is a bit confusing to me. Given that the same commands are used to run Percy tests & all the settings are the same, why would there be a difference in width/viewport? 🤔 My understanding was that they should be the same.

Thanks again!

[Robdel12]

I wish I could do this sort of debugging locally

I'm hoping we introduce debugging tools this year -- it's been coming up a bunch in planning (I've been pushing hard for it). You could use the download HTML option, but admittedly it's not very useful (it's just HTML, no assets)

why would there be a difference in width/viewport?

I'm not sure, Percy doesn't have any insight into that. That's the Ember test runner / application / test setup -- we're just capturing the exact state of the DOM when percySnapshot is called. Sometimes it's something like scrollbars present locally vs not in CI (or vice versa). That could cause a 10px diff.

[twokul]

@Robdel12 I re-ran GH action using a different OS (macOS this time.) and all the differences in SVG text width & positioning are gone. The only difference that I see is font rendering which I guess makes sense? Build Link.

Is there a difference between how SVG is rendered by Chrome on Linux vs. Mac?

[github-actions[bot]]

This issue is stale because it has been open for more than 14 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 28 days with no activity.

[mixonic]

@Robdel12 and team is there a follow-up possible on the mac vs. Linux environments?

[wwilsman]

Hi @mixonic!

If I'm following correctly, is this your question?

Is there a difference between how SVG is rendered by Chrome on Linux vs. Mac?

I don't think this question has anything to do with Percy, but rather a general browser/platform question. I'll try my best to answer though. 🙂

In my own experience, the main differences between platforms will be fonts and possibly the presence of scrollbars. SVGs themselves should be rendered more or less consistently as they are simply scalable images. The issue pointed out above shows that a dynamic width is changing between snapshots. Those width attributes wouldn't change by themselves as SVG files usually contain static content. But instead, that dynamic width is likely caused by whatever is inserting, creating, or manipulating the SVG on the page.

As @Robdel12 commented before, we're just capturing the exact state of the DOM when percySnapshot is called. How that DOM is rendered before it is captured will be entirely out of Percy's control and up to the app/site. For example, if you forced that SVG width to be a specific value right before taking the snapshot, that's the width that would be captured by Percy.