search

w3c / csswg-drafts

CSS Working Group Editor Drafts
https://drafts.csswg.org/
Other
4.5k stars 661 forks source link

Update CONTRIBUTING.md to match how the testing process really works #2503

Open foolip opened 6 years ago

foolip commented 6 years ago

In the conclusions I wrote:

CSS specs came up three times. The WG decision → spec update → Needs Testcase (WPT) process means that tests aren’t generally expected to be written at the time of the spec change and in our investigation we normally dropped in between the “spec update” and “test written” stage, and we haven’t looked into when the tests normally end up being written. Whatever the process is, it’s not what is described in CONTRIBUTING.md.

CONTRIBUTING.md was updated in https://github.com/w3c/csswg-drafts/pull/1767, see that PR for background.

I think CONTRIBUTING.md now matches neither how things actually work or the aspiration, in particular "Typically, both PRs will be merged at the same time" doesn't make sense since spec changes aren't done via PRs and the test updates normally don't happen at the same time.

I think the current process is to close all "Needs Testcase (WPT)" issues for spec before publishing it as CR or higher next time, but better that someone who's in the loop spells it out than me guessing.

foolip commented 6 years ago

Oops, I forgot to add the intro:

Last quarter @rakuco and I worked on "Find and investigate [something] every week" once a week to see how testing is working in practice. The most important part of it was "A normative spec change without tests".

frivoal commented 6 years ago

I think the current process is to close all "Needs Testcase (WPT)" issues for spec before publishing it as CR or higher next time, but better that someone who's in the loop spells it out than me guessing.

I believe this is indeed what the process is intended to be. I don't know if we have been running it long enough to have enough CRs to confirm that it works, and that we're not accumulating an excessive backlog of "Needs Testcase (WPT)".

foolip commented 6 years ago

@frivoal, what do you think the best time to look for existing test cases is? The simplest answer is at the same time as resolving the "Needs Testcase (WPT)" issue, but that will leave a (possibly long) window during which the tests contradict the spec. I haven't yet noticed any confusion or trouble because of this happening, and it cannot be eliminated entirely because finding tests can be very hard, but is it a problem worth trying to solve?

frivoal commented 6 years ago

As soon as "Needs testcase (WPT)" is added is a good time for third party (i.e. anybody other than the editors) to check if we actually already have test cases covering that and submit some new ones if we don't.

When we try to transition to CR is a good time for chairs to check if there are any "Needs testcase (WPT)" labels left.

Editors who maximize for throughput will typically hope that someone else writes the test, focus on spec changes, and (maybe) try to plug the remaining holes when we try to transition the spec.

Editors who maximize for the velocity of each individual change will be more proactive in write tests and reporting upstream bugs.

If you want to be starting your search with normative changes, mostly you'll find that almost none of them (initially) have test cases, so the best thing to do is probably to check that there is a corresponding "Needs testcase (WPT)" label and add it if it's not there, so that someone follows up eventually.