search

w3c / modern-tooling

Work of the modern tooling task force
http://w3c.github.io/modern-tooling/
MIT License
44 stars 39 forks source link

Testing, Publishing results #13

Closed r12a closed 9 years ago

r12a commented 9 years ago

these are a few thoughts off the top of my head related to that section...

i'd like to see the testing framework changed so that we can indicate partial passes where appropriate, and capture notes about why so labelled (see http://www.w3.org/International/tests/repo/results/text-transform#upperlower for an example where this comes in handy).

i'd also like us to be able to reflect support per browser, rather than just per engine The i18n test framework allows you to specify the browser, and it's quite useful. There are some tests that yield different results even though they are based on the same engine. This would also enable our results to be used to update pages such as MDN.

i think we should control carefully who can submit test results and make it easy to quickly fix incorrect submissions. Just yesterday i was talking with someone about bidi tests and looking at the shepherd interface just got us confused and wasted time checking tests until we ascertained that the results displayed were not really reliable. (We went back to the i18n results page instead - at www.w3.org/International/tests/repo/results/writing-modes-bidi).

it would be really nice to make the system in the CSS specs more widely available, whereby each section links to tests that test features within it.

test results should have a way to indicate dependency paths. See an example at http://www.w3.org/International/tests/repo/results/hyphens where some tests are only valid if others work, and that is documented.

not sure whether those are useful ideas, but i thought i'd share just in case

plehegar commented 9 years ago

Seems like a discussion for public-test-infra or https://github.com/w3c/wpt-tools ...

darobin commented 9 years ago

These are useful ideas, but I think they're too detailed for this document. I agree with @plehegar that they would likely be better processed as part of the ongoing development of the WPT project.

The WPT test runner could easily be enhanced to support partial passes with notes, and wpt-report could expose that. It's probably worth filing bugs there (but starting with a mailing list discussion is probably better — I can file the relevant bugs afterwards).

Expressing dependencies between passes might be more contentious; but I think you should make the case. (It is probably a good idea to bring these ideas up in separate emails.)

I'm closing the issue here, but the ideas shouldn't be lost. If you don't have the time to bring them to test-infra please let me know and I will.