percy / percy-cypress

Visual testing with Cypress and Percy
https://percy.io
MIT License
340 stars 41 forks source link

Percy-Cypress v3 does not load fonts #312

Closed khitrenovich closed 3 years ago

khitrenovich commented 3 years ago

We are trying to migrate one of our projects to the new Percy-Cypress v3, and we have problems with custom fonts not being rendered on the Percy screenshots.

Sample, with differences highlighted: image

Sample, with no highlights: image

From debug logs, I see that font files (woff and ttf) are being loaded -

[percy:core:discovery] Handling request for http://localhost:8080/fonts/CustomSansTTRegular.woff2 (42ms)
...
[percy:core:discovery] Handling request for http://localhost:8080/fonts/CustomSansTTRegular.woff2 (17ms)
...
[percy:core:discovery] Processing resource - http://localhost:8080/fonts/CustomSansTTRegular.woff2 (4ms)
[percy:core:discovery] Making local copy of response - http://localhost:8080/fonts/CustomSansTTRegular.woff2 (5ms)
[percy:core:discovery] -> url: http://localhost:8080/fonts/CustomSansTTRegular.woff2 (1ms)
[percy:core:discovery] -> sha: b0bacf06b2e7a3e3fc672625c9aaff825ed79d754a174c0f352d68d10daf7397 (0ms)
[percy:core:discovery] -> filepath: /tmp/percy/b0bacf06b2e7a3e3fc672625c9aaff825ed79d754a174c0f352d68d10daf7397 (1ms)
[percy:core:discovery] -> mimetype: font/woff2 (0ms)

...

(in the following tests)
[percy:core:discovery] Handling request for http://localhost:8080/fonts/CustomSansTTRegular.woff2 (1ms)
[percy:core:discovery] Response cache hit for http://localhost:8080/fonts/CustomSansTTRegular.woff2 (1ms)

All that looks pretty legit, besides the duplicate "handling request" line. Yet, the screenshots are rendered with the default font.

Any idea what went wrong here? Are we missing any pieces in the migration process?

Robdel12 commented 3 years ago

Hey @khitrenovich! This seems to be a bit more support related but we can help here if you are willing to share a build link.

My first question though: how are you serving these fonts? Wouldn't happen to be with http-server, would it?

khitrenovich commented 3 years ago

It's in a private repo, so can't share a link, unfortunately.

To your question - The fonts are coming from the same server as the React app itself, which is running locally. Our CI is running in GitHub Actions. Cypress e2e tests are triggered via yarn e2e command.

Relevant scripts -

    "start:light": "webpack serve",
    "cy:run": "percy exec -- cypress run --browser chrome",
    "e2e": "start-server-and-test start:light http-get://localhost:8080 cy:run"

Percy config:


version: 2
snapshot:
  widths: [660, 1280]
  percy-css: |
    .dropdown__menu > div:nth-child(1) {
      background-color: #39393b;
    }
    .dropdown__menu > div:nth-child(3) {
      background-color: #39393b;
    }

I'll be glad to provide whatever info I can to solve that. Thank you!

Robdel12 commented 3 years ago

Gotcha! You'll have to go through support@percy.io to send a build link.

The fonts are being captured -- my guess is they're not being served with the correct encoding (from what I've seen in the past). We'd have to download the snapshot to find out what was captured for those fonts.

Is this package in your dep tree at all? https://www.npmjs.com/package/ecstatic Basically could be happening ecstatic is serving those fonts with the wrong encoding. If that's the case it isn't something we can fix (since it can be replicated with Chrome & that web server -- completely external to Percy).

khitrenovich commented 3 years ago

We are using webpack serve, which actually wraps webpack-dev-server. There is no express involved.

Re fonts encoding - The setup we have works fine with the latest Percy-Cypress v2.x - Had something changed in the area in v 3?


Can I send you percy.io build link in a DM somehow? I apologize for trying to solve that here. My recent experience with Percy support (since it was moved under BrowserStack) is totally disappointing - and we are paying Percy user!

Robdel12 commented 3 years ago

The only thing that changed is the version of Chromium that asset discovery. We dropped Puppeteer and download a specific version of Chromium now (which probably changed from what was being used by the old toolchain). I've noticed Chrome 85+ at least have this issue (we're using a rev of 87: https://github.com/percy/cli/blob/master/packages/core/src/install.js#L47-L53 close to what's in Percy's env).

You could drop a snapshot ID here? That's all I'd need to download the snapshot (internal only facing tool)

khitrenovich commented 3 years ago

@Robdel12 Build 9750416, snapshot 549314950 (or any other from that build). Thank you so much!

Robdel12 commented 3 years ago

Hmm, yeah it looks like these fonts are captured as corrupt.

image

When I debugged this in the past, I was able to replicate by starting the same server with correct fonts and visit the page in Chrome. In the network debugger when I selected the font, the preview was clearly not the same font. Doing the same thing, but starting a simple python server in the directory instead, served the fonts correctly. I wish I could share this, but it was with a customer's snapshot. You could try and reproduce this or drop a link to a known working/clean set of fonts and I can.

With broken fonts, you'll see the preview render incorrectly: 2021-04-08 14 01 18

Something between the server serving the woff2 fonts & Chrome's network layer has issues. And since we use Chrome for network interception, we're impacted by this too. The interesting thing now is, we previously thought it was isolated to http-server, but maybe not?


FWIW, I bet @percy/agent breaks the same way if you were to update agent to the latest & puppeteer to the latest (so it downloads the latest chrome revision it needs). I'm 95% sure this is related to the version of Chromium being used for network interception (asset discovery). The inverse also works here too! Setting the asset discovery browser to an older version of Chromium likely will "fix" the issue (more like kick it down the road)

khitrenovich commented 3 years ago

@Robdel12 An example of the working build (one with Percy v2) is 9775466, snapshot 550603029.

vdineva commented 3 years ago

I am seeing the same font loading issue in @percy/cypress@2.3.4

Robdel12 commented 3 years ago

Curious -- if you modify your app (temporarily to test) to only serve woff (not woff2), are the fonts no longer corrupt/appear in snapshots?

khitrenovich commented 3 years ago

Curious -- if you modify your app (temporarily to test) to only serve woff (not woff2), are the fonts no longer corrupt/appear in snapshots?

@Robdel12 Here it goes - build 9794445, snapshot 551699390, same visual effects.

Robdel12 commented 3 years ago

Would you be able to post your webpack config / server config? Or a small project that reproduces it? I'm not able to replicate with a bare bones / fresh setup.

Also going to cc @wwilsman since I'm OOO today.

khitrenovich commented 3 years ago

I've just posted the webpack config here. Let me know if you still need the project after looking at the config, and I'll do my best to find some time for that in the coming days.

khitrenovich commented 3 years ago

@Robdel12 / @wwilsman - Do you have any update on that, please?

khitrenovich commented 3 years ago

FYI - Upgrade to v3.1 and CLI 0.48 brought no changes.

Robdel12 commented 3 years ago

I'd expect that. :D CLI uses a very specific version of Chrome (and we change it manually -- different from @percy/agent which changes Chrome under the hood as Puppeteer does, that causes a lot of issues for us in SDKs like this).

No updates here -- this issue has been around since at least January (which predates CLI & existed in @percy/agent). The last time this was "fixed" we had a solid reproduction provided and it was due to the server serving Mojibaked responses of the fonts. So by the time Percy captured the request from Chrome, it was already corrupt. Nothing Percy's SDK could do there. This looks exactly the same, except sadly it's not http-server's fault. We fixed it by not using http-server, since it was serving broken assets to Chrome.

My guess is this happens because the server & the version of Chrome. Unfortunately this is not easy to replicate, so having that solid replication would make this more solvable.

khitrenovich commented 3 years ago

We were able to isolate the issue to a combination of webpack-dev-server and file-loader. Our solution was to use url-loader instead of the file-loader with webpack-dev-server - that is, locally and in PR builds in CI. You can add one more recipe to your collection of the workarounds. Thank you for helping us to solve this mystery!

Robdel12 commented 3 years ago

Oh man, that is so rough. I wish this wasn't so awful to debug! Thanks for that info, it does make me wonder if keeping the same setup & building as production would also fix the issue.

Local dev servers tend to do interesting things to assets in the name of reload-ability (like serve CSS as blob asset, for faster live reloads). This might be another one of those quirks 😫