web-platform-tests / wpt

Test suites for Web platform specs — including WHATWG, W3C, and others
https://web-platform-tests.org/
Other
4.88k stars 3.05k forks source link

Tests relying on larger viewports #5403

Open gsnedders opened 7 years ago

gsnedders commented 7 years ago

The documentation states variously that reftests are taken at 600x600 and 800x600; they're mostly run at 600x600 and this apparently matches what was decided for the CSS testsuite years ago so let's just go with that.

Note we also have a <meta name='viewport-size'> that's totally undocumented and that only works for Servo per w3c/wptrunner#166.

This follows on from #4619 but provides a far better list of affected tests and includes the now merged CSS testsuite.

mrego commented 6 years ago

Just curious, how did you get the numbers?

gsnedders commented 6 years ago

@mrego a script that I thought I had on gist.github.com somewhere but seemingly not; essentially going through all reftests and reading something like the offsetHeight of the root element.

mrego commented 6 years ago

It'd be really nice to have it around, I realized that I was using 800x600 (that's what Chromium and WebKit use) instead of 600x600 in #9716.

gsnedders commented 6 years ago

@mrego https://gist.github.com/gsnedders/6721716f6f90380ac6b76acbdbee91a2; I have no idea whether we'll standardise on 800x600 or 600x600, FWIW.

mrego commented 6 years ago

Hey, thanks for the link. I'll give it a try at some point.

So I thought 600x600 was agreed before, if not what I have seen is that Chromium an WebKit use 800x600, dunno what happens in Firefox or Edge. If everybody uses already 800x600 that could be the right value to standardize.

gsnedders commented 6 years ago

@mrego Firefox just uses wptrunner so it does use 600x600. I have no memory of what Edge does (@thejohnjansen?).

TalbotG commented 6 years ago

1- So the definitive standard for tests is (should be?) 600px wide by 600px tall?

Test assumptions now state:

The device has a viewport width of at least 800px.

2- A lot of tests could be retrofitted to comply with 600x600. In some tests, that could be more difficult or a lot more difficult ...

3- Tests that occupy more than 1000 (1 thousand) in width or height should be examined or reconsidered ... :

css/CSS2/abspos/abspos-containing-block-initial-001.xht (588x10110) css/CSS2/bidi-text/line-breaking-bidi-001.xht (3608x588) css/CSS2/bidi-text/line-breaking-bidi-002.xht (3608x588) css/css-backgrounds-3/border-radius-clipping.html (160008x160048) css/css-flexbox-1/flex-direction-with-element-insert.html (588x1243) css/css-multicol-1/multicol-fill-auto-block-children-001.xht (1208x588) css/css-multicol-1/multicol-inherit-003.xht (1208x588) css/css-multicol-1/multicol-inherit-004.xht (1208x588) css/css-regions-1/counters/extract-list-items-012.html (588x1180) css/css-regions-1/counters/extract-ordered-lists-in-regions-001.html (588x1317) css/css-regions-1/counters/extract-ordered-lists-in-regions-002.html (588x1156) css/css-regions-1/counters/extract-ordered-lists-in-regions-003.html (588x1200) css/css-regions-1/counters/extract-ordered-lists-in-regions-explicit-counters-001.html (588x1501) css/css-regions-1/counters/extract-ordered-lists-in-regions-explicit-counters-003.html (588x1335) css/css-regions-1/counters/extract-ordered-lists-in-regions-explicit-counters-004.html (588x1373) css/css-regions-1/counters/extract-ordered-lists-in-regions-explicit-counters-005.html (588x1392) css/css-regions-1/counters/extract-unordered-lists-in-regions.html (588x1317) css/css-regions-1/writing-modes/extract-ordered-list-direction-and-writing-modes.html (588x1544) css/css-text-3/text-transform/text-transform-capitalize-001.html (588x1636) css/css-text-3/text-transform/text-transform-capitalize-003.html (588x3476) css/css-text-3/text-transform/text-transform-capitalize-005.html (588x1716) css/css-text-3/text-transform/text-transform-capitalize-007.html (588x2516) css/css-text-3/text-transform/text-transform-capitalize-010.html (588x1796) css/css-text-3/text-transform/text-transform-capitalize-011.html (588x1156) css/css-text-3/text-transform/text-transform-capitalize-014.html (588x1876) css/css-text-3/text-transform/text-transform-capitalize-016.html (588x3076) css/css-text-3/text-transform/text-transform-capitalize-018.html (588x3796) css/css-text-3/text-transform/text-transform-fullwidth-001.xht (588x12415)

etc...

This one css/css-writing-modes-3/flexbox_align-items-stretch-writing-modes.html (17895698x588) certainly looks suspicious...

gsnedders commented 6 years ago

@TalbotG

So the definitive standard for tests is (should be?) 600px wide by 600px tall?

See #7598; we're not consistent here, though the upstream maintained runners use 600x600. It seems likely that we'll change that to be larger, however.

A lot of tests could be retrofitted to comply with 600x600. In some tests, that could be more difficult or a lot more difficult ...

That's definitely true. That said, there's definitely an argument as to whether it's a worthwhile use of resources to fix these tests versus other things people could work on, especially if we do change the (default) size.

Tests that occupy more than 1000 (1 thousand) in width or height should be examined or reconsidered ...

From a brief glance, they're mostly either deliberately doing that or ex-Presto tests (which were 1600x1200).