css-meeting-bot
commented
8 months ago The CSS Working Group just discussed republishing Grid and Flex, and agreed to the following:
RESOLVED: Change CRD requirement for tests from MUST to SHOULDRESOLVED: Republish CRD of grid/flex after fantasai's auditACTION: emilio to file an issue about improving automation to comment on the issues
The full IRC log of that discussion
<astearns> topic: republishing Grid and Flex<emilio> github: https://github.com/w3c/csswg-drafts/issues/6900
<emilio> fantasai: grid specs have not been published in 4y, flexbox even more (2018)
<emilio> ... from a process perspective this is pretty bad
<emilio> ... means that our official spec is massively out of date
<emilio> ... it'd be great to get to a point where the folks looking at drafts is us while we discuss recent changes and get them published
<emilio> ... we need to review the changes and tests, and write the tests necessary to make sure that every change has a test
<fantasai> s/drafts/editors drafts/
<fantasai> s/the folks/the only folks/
<emilio> ... we can wait for me and tab to do that which will probably take a while
<emilio> ... we can assign someone else which was supposed to happen but didn't
<emilio> ... or we can re-publish without the test-case requirement
<florian> q+
<emilio> ... we can block the CR on having tests, but at least TR will be up-to-date
<astearns> ack florian
<fantasai> s/CR/CRS/
<emilio> florian: I think we should republish at this. This is a lot of work for specs that I edit, and I do it for very simple specs
<fantasai> s/very/comparatively/
<emilio> ... for these it's a lot of work
<emilio> ... the test requirement is not a process requirement, it's a working-group requirement
<florian> q+
<astearns> ack florian
<emilio> astearns: I'm willing to make an exception for this publication, but I think I still like the idea of requiring tests for post-CR changes
<emilio> TabAtkins: evidence suggests the rule is not being followed
<emilio> florian: we _might_ have tests, part of the issue is that we don't have a way to map the tests to the section of the spec they test
<emilio> iank_: there's a link to the spec
<emilio> florian: it takes me more time to figure out what the test is testing that everything else
<emilio> rachelandrew: I agree, it's a hell of a job, takes a lot of time to go through them, very grateful for people that put notes on what the test is testing
<emilio> q+
<iank_> q+
<emilio> florian: whether of not we change the culture of testing or requiring more documentation
<fantasai> ^rachelandrew: can take 30-45 minutes per test to even figure out what it's testing
<emilio> ... as it is right now, finding what tests are testing is a lot of work
<astearns> ack emilio
<emilio> ... for comparatively small specs it's manageable, for big complex specs is not
<fantasai> emilio: We have a lint that guarantees every test in CSS directory has a link to the spec
<fantasai> emilio: but you can drop a link to the spec, and not to the relevant section
<fantasai> emilio: we could be more strict about what acceptable links are
<astearns> q+
<fantasai> emilio: or maybe require an assert metatag or title tag to explain what it's doing
<fantasai> emilio: not to object to this publication, we should not have 6-year-old specs on /TR
<fantasai> emilio: but if it saves editors time, it is true when I write tests, the only thing I want is to get the test to work wrt the patch i am writing
<fantasai> emilio: I land the tests to move on
<fantasai> emilio: but more than happy to add spec link, can also write more words to help the editors figure out what the test is testing
<fantasai> emilio: generally I try to both spec and bug link when I write tests, so it can be easy to follow the link
<fantasai> emilio: but maybe we should require a little bit of description
<astearns> ack iank_
<emilio> iank_: one thing I want to point out is that it's likely that implementors don't really care there's duplicate tests
<emilio> ... adding a test is less effort than finding an existing test
<astearns> ack fantasai
<emilio> astearns: proposal: for specs in CR+ to not /merge/ a change without a test
<florian> q+
<emilio> TabAtkins: no way, that'd slowdown working on CR specs a lot
<astearns> ack astearns
<emilio> astearns: maybe the slowdown is less at the time you're making the change than finding all of WPT afterwards
<emilio> dbaron: let me take one step back
<emilio> ... one of the motivations is that there should be a moderately high bar to change specs that are in CR
<emilio> ... because they're implemented etc
<emilio> ... a lot of the discussion has gone around finding through tests
<emilio> ... the alternative workflow is, when we have a post CR check, we make sure that we have links from the PR to the browser bugs
<emilio> ... and those bugs should end up having tests
<emilio> ... there's a different case which is changing a CR to reflect implementations
<emilio> ... a potential way forward would be to split CR changes in two:
<emilio> ... * changes that reflect what is already implemented: try to get the test checked in when the change is
<emilio> ... * changes that require implementation changes: Make sure that implementation bugs are linked from the issue, then finding the tests is trivial by looking at impl bugs
<emilio> q+
<emilio> ... I think that makes the problem a bit simpler
<emilio> ack dbaron
<astearns> ack florian
<emilio> florian: one thing that bothers me (searching tests / writing tests / linking) is always work for the same people
<emilio> ... the idea that other people would do that doesn't happen in practice
<astearns> ack emilio
<emilio> ... if it's more work they have to do then that's great if they have infinite time but they do not
<emilio> ... so maybe the rest of the WG has to step up and maintain more specs
<astearns> +1 to more editors to spread out the work
<emilio> ... or we require the editors to do that and recognize that they would edit fewer things
<castastrophe> q+
<fantasai> [the fork above was, if there's a total amount of work that's more, dumping them on overworked people doens't work, so either the rest of the WG needs to help with the testing stuff or needs to take on more editing stuff or more stuff gets dropped on the floor]
<fantasai> emilio: at least Chromium and Gecko have automation to track WG resolutions that link back to the original issue
<fantasai> emilio: that helps make sure that the resolutions get linked to browser bugs
<fantasai> emilio: we do the work of triaging and opening bugs as needed already
<fantasai> emilio: maybe could use that as a way to figure out the lins
<fantasai> emilio: I don't think filing browser bugs in every issue tracker should be responsibility of Elika and Tab
<fantasai> emilio: we already have a bot to track the resolutions
<florian> s/to step up and maintain more specs/to step up and help with the extra steps we require
<fantasai> emilio: we could make more explicit where the browser bugs are, or maybe our bot could comment directly in the issue or something
<fantasai> emilio: WHATWG/HTML require when you open a PR to have implementation bugs and link from the description
<florian> s/and recognize that they would edit fewer things/and recognize that they would edit fewer things and then the WG needs to step up and edit more things
<fantasai> emilio: I think that's reasonable compromise, especially if Elika and Tab don't have to do it
<fantasai> emilio: I agree with dbaron that finding tests would be easier if we go through implementation bug reports
<fantasai> emilio: let's discuss what we can do to help editors
<fantasai> emilio: obviously more editors, but, besides that
<astearns> ack dbaron
<florian> q+
<fantasai> dbaron: basically just that, we should revise the bots that we use so that they comment back on CSSWG issue to point to the relevant browser issues
<fantasai> astearns: people could remove the Needs Testcase tags once there's tests checkd in
<florian> fantasai: we have "needs test case" as well as "tested" labels
<fantasai> castastrophe: I wonder if we have a backlog, if we can breakdown what the backlog is and request funding for temporary contractors?
<fantasai> florian: I've done it on some specs as a contractor, but not those specs
<emilio> fantasai: proposal is to drop the tests for CRD, but keep it for CRS
<emilio> emilio: CRD?
<emilio> fantasai: CR Draft doesn't have the checks for the patent policy etc... lots of stuff is checked for CRS
<emilio> ... we could fork that and at least that way
<emilio> ... not great that we haven't published a snapshot in 6 years but better than not publishing a draft in 6 years
<fantasai> s/patent policy etc/wide review, addressing FOs, etc, and also has full patent protection/
<emilio> astearns: so proposal is changing the requirement for CRD to _should_ have tests
<emilio> astearns: objection?
<emilio> dbaron: as long as it's clear that CRS keep that requirement
<dbaron> dbaron: relative to the previous CR or CRS
<emilio> RESOLVED: Change CRD requirement for tests from MUST to SHOULD
<emilio> fantasai: have to do some cross-checking that they're all in sync and prepare grid/flex for publication
<emilio> ... not ready today but can audit and bring back to the wg
<emilio> RESOLVED: Republish CRD of grid/flex after fantasai's audit
<florian> q+
<astearns> ack castastrophe
<oriol> q+
<emilio> ACTION: emilio to file an issue about improving automation to comment on the issues
<astearns> ack florian
<emilio> florian: q about automation
<emilio> ... we'd generate a link from GH to browser bugs
<fantasai> ... could we also add the has tests label if the browser bug closes?
<fantasai> emilio: could maybe do
<fantasai> dbaron: [suggests some automatic stufff]
<matthieud> q+
<astearns> ack oriol
<dbaron> Step 1 is making sure browser bugs are linked from the CSSWG issue. Step 2 is linking that to the automation that ties those browser bugs to WPT PRs -- probably commenting on CSSWG issue when WPT PRs are created and maybe changing labels when they're merged.
<emilio> oriol: I wanted to comment on automation, sometimes I did create it, pointed out to the spec, and nobody implemented it in the issue but was fixed somewhere else
<emilio> ... like a year later somebody comes around but that link is lost
<astearns> ack matthieud
<emilio> ... we should make sure that the issues created by these bots are not closed blindly but have the right references to what fixed it
<emilio> matthieud: can't the test be written when the change is made?
<emilio> florian: Ideally yes but can be a lot of work for the editor
<fantasai> emilio: everyone agrees that having tests ready with the spec change, and even fixing existing tests, would be great
<fantasai> emilio: but that's a lot of work for the editors
<fantasai> emilio: as an implementer, when you fix the behavior, it's easy to see which tests need to be fixed
<fantasai> emilio: or if tests need to be added
<fantasai> emilio: so in practice -- and especially since Tab and Elika do 50% of spec editing -- it's not practical for the editors to do all the testing work for the spec changes
<fantasai> emilio: we want to reduce the workload for publishing
<castastrophe> a half-joking suggestion that we train an ML model to write and review tests :wink:
<fantasai> emilio: the linking work here will help implementers, too
astearns
svgeesus
fantasai
myakura
frivoal
noamr
Tag the IRC bot with this thread whenever publication resolutions are being discussed.
Agenda+ and add a comment if you want to publish something.
See also Estimated Publication Badness Chart and Transition Requests
Latest:
Backlog of Uncertain Blockedness: