search

web-platform-tests / wpt.fyi

web-platform-tests dashboard
https://wpt.fyi/
Other
188 stars 89 forks source link

Reftest analyser sometimes shows zero diff of failing test #3276

Open gsnedders opened 1 year ago

gsnedders commented 1 year ago

Following the link from https://wpt.fyi/results/css/CSS2/borders/border-left-color-132.xht?run_id=5142037970550784&run_id=5093854074634240 results in:

maxDifference: 255 totalPixels: 479972

… which explains the fail. But only sometimes. Otherwise it shows no difference. No idea what's up with this.

gsnedders commented 1 year ago

FWIW, this shows what I'm seeing:

Screenshot 2023-04-26 at 18 10 04

I'm starting to wonder if it believes almost every pixel on the actual screenshot is RGB(0, 0, 0)… which also seems wrong.

And I'm getting different results in every browser. Firefox Nightly says:

maxDifference: 0 totalPixels: 0

Chrome Canary says:

maxDifference: 1 totalPixels: 18432

And STP says:

maxDifference: 255 totalPixels: 479972