tc39 / test262

Official ECMAScript Conformance Test Suite
Other
2.31k stars 459 forks source link

Add metadata about the ECMAScript edition that each feature was introduced in #4161

Open jedel1043 opened 1 month ago

jedel1043 commented 1 month ago

It's pretty common for test runners such as test262.fyi to have a way to filter tests per ECMAScript edition, and some people like to analyze the conformance of engines per ES version.

Currently some runners have opted for using "features.txt" to extract this information, since features have a clear edition where they were introduced, and this could be used to know which minimum edition a test supports. However, there's some duplicate effort around this, since all runners have to maintain a separate mapping from the feature to the corresponding edition that introduced it:

Seeing this, it would be nice to remove this duplicate effort by moving this metadata here.

Possible points of discussion

ljharb commented 1 month ago

I think a unified file would be nicer, but that might interfere with how existing runners interface with the existing one.

(also, we should be using years, post-ES6, not edition numbers)

jedel1043 commented 1 month ago

I think a unified file would be nicer, but that might interfere with how existing runners interface with the existing one.

This is possible to do using comments. The file already has some comments, so we could probably leverage that to add metadata in a backwards-compatible way.

jedel1043 commented 1 month ago

(also, we should be using years, post-ES6, not edition numbers)

That's fair. I'm just using the edition identifier used by the two example runners, but it would also be understandable to use the more "official" identifier instead.

(runners could always subtract 2009 if they need the "semver" identifier)

PawelJastrzebski commented 1 month ago

Added concept of toml file in the PR.

ljharb commented 1 month ago

I'm not sure toml is a good choice solely because it's not natively parseable in node and other platforms (altho i love toml!)

PawelJastrzebski commented 1 month ago

That's a valid concern. Considering wider support, using a JSON file seems like a better choice, despite sacrificing some of the descriptive commentary.

added .json example to the PR

jedel1043 commented 3 weeks ago

That's a valid concern. Considering wider support, using a JSON file seems like a better choice, despite sacrificing some of the descriptive commentary.

I think using YAML would be the most consistent in this case. Every test needs YAML in order to be interpreted correctly, so I think we should avoid introducing another metadata format.

linusg commented 3 weeks ago

Given you quoted test262.fyi as a use case I would strongly advise against YAML, browsers can't parse that :)

There are also languages that still don't have a decent YAML parser (for instance Zig), because YAML is an awfully complex format. I consider test262 using YAML metadata a mistake that's hard to rectify at this point, but please don't use that as an argument to add more YAML.

jedel1043 commented 3 weeks ago

@linusg JSON is also fine by me. The one thing I would advocate for is to keep everything on the features.txt file by using the comments, instead of having two separate files.