search

rust-lang / crates.io

The Rust package registry
https://crates.io
Apache License 2.0
2.98k stars 601 forks source link

Spurious Percy test failures #9052

Closed LawnGnome closed 4 months ago

LawnGnome commented 4 months ago

Many recent Percy builds seem to have failed due to it not loading the Fira fonts that crates.io uses, which causes visual changes when the fallback fonts are used.

Example: https://percy.io/b1b3bf45/crates.io/builds/35231632/

Turbo87 commented 4 months ago

this is caused by us currently running the original frontend test suite, and the playwright test suite in parallel. these both share the same Percy snapshot names, and one loads the webfont, the other doesn't.

tl;dr it's a race condition 🙈

/cc @eth3lbert

eth3lbert commented 4 months ago

It might be better to place it in a separate Percy folder (project?) if possible. Would that be a valid option for us?

Turbo87 commented 4 months ago

hmm, I think there is a 1-to-1 correlation between Percy projects and GitHub projects, so making it a dedicated project would be difficult, but I guess we could prefix/suffix the playwright snapshots accordingly?

eth3lbert commented 4 months ago

but I guess we could prefix/suffix the playwright snapshots accordingly?

Oh, this should also work.

Then, I believe we only need to adjust the title here https://github.com/rust-lang/crates.io/blob/cb34f1974ffab39e4dde0e0871769dba1886a33c/e2e/fixtures/percy.ts#L13-L18, which I intentionally implemented with the same name as the @percy/ember one for consistency.

Turbo87 commented 4 months ago

it looks like https://github.com/rust-lang/crates.io/pull/9054 has fixed this :)