PolariTOON
commented
2 years ago Is that definition of what's included in a CSS snapshot filter ok?
212 defines an unique filter with several choices, including one choice per CSS Snapshot.
As i said in the PR, each snapshot has a significant overlap with its previous / next one, and most of the features are known to be largely supported since they are included in the official language definition. This results in a lot of green results for all snapshots, that may hide what was added in each version and may not be entirely implemented by major browsers yet.
Should there be difference filters for snapshots? I.e. show only specifications that got added in a snapshot and exclude those that are already included in a previous snapshot? If so, how should those be integrated?
For the reasons given above, here's a proposal: i imagine, for each snapshot, that we could only include the diff (what was added) with the previous snapshot. This would make new stable features and the corresponding tests for them easier to find. In fact, that's what has always been done with all the spec levels for now: to not test again features that were redefined in a superseding spec.
Then, we could have one checkbox per snapshot, all of them independent, allowing to have both the diff behavior by checking only one of them, or the behavior introduced in #212, by checking all the checkboxes up to the desired snapshot.
Should there also be filters for specifitions defined as "fairly stable" and "roughly interoperable" in the snapshots? (See https://www.w3.org/TR/css-2021/#fairly-stable and https://www.w3.org/TR/css-2021/#rough-interop.) If so, how should those be integrated?
"fairly stable" and "roughly interoperable" could have their own filter choice / checkbox next to the snapshots, but perhaps we should only make them according to the latest snapshot to avoid having some specs appear in several filters. Since we already use to test these modules in this project, just adding a way to filter them should not have a big cost.
Also "CSS2", "SVG" or WhatWG specs could get their own filter, as @Zefling suggested there: https://github.com/LeaVerou/css3test/pull/212#issuecomment-1006495331 And we could put the rest in a "Others" category in order to make all the categories be a partition of the set of specs.
That way, if we decide to use checkboses instead, checking all the checkboxes would be equivalent to the current filter "all". We would have to discuss what should be the default state: should CSS2 / unstable specs / non-CSS specs be included by default?
Should "safe to release" features be included in the filters? (See https://www.w3.org/TR/css-2021/#CR-exceptions.) If so, how should those be integrated?
"Safe to release exceptions" are an interesting case because there are not included on a module basis but on a feature basis. They currently may be selectors, properties, or even values, but this may evolve. To specify they are exceptions, i only see think about two options: we have to (1) mark each of them independently, or (2) list all of them somewhere.
For (1), we would have to change the current structure of the test file, since some of the exceptions tested are not objects but only strings (for example, min-content) and thus cannot be annotated as is.
For (2), we would need to define some way to declaratively reference a given test somewhere in the testbase.
Either way, this seems more complicated to me that the other changes needed for my proposal, so if we really want to integrate them, i think this should come later.
Does this path (having disjoint filters, being able to select multiple of them, and then integrating "safe to release" exceptions") appear desirable and realistic to you?
SebastianZ
The addition of filters in #212 introduced a way to define the first and last CSS snapshot a specification is part of the official CSS definition. This allows to filter the list of specifications by exactly the ones that are part of the definition of CSS in a specific snapshot and excludes all others, also specifications that are superceded by others.
That was the simplest solution I could come up with. Though there may be other ways to define the filters related to the snapshots.
Specific questions:
Sebastian