Testing with Test262

[jugglinmike]

Test262 is the official test suite of the JavaScript language. Although its primary use case is JavaScript run times, its extensive coverage of the language makes it useful for testing other kinds of interpreters. This is especially true for new language features since tests are a formal requirement at an early stage in the language standardization process. That leads to very good support for new language features.

In recent years, I've been working to integrate Test262 in the test suites of various JavaScript parsers. I'm interested in doing the same for lexing projects like Atom's syntax highlighter. Basically, we would attempt to lex each test file and assert that only a subset produced errors.

Right now, Test262 doesn't have the proper metadata to support this use case. It describes "negative syntax" tests generally without distinguishing grammar errors (e.g. var class) from lexing errors (e.g. "unclosed string). I'd be excited to introduce that information, but I'd like to gauge interest here, first.

Would the maintainers be interested in using Test262 like this?

[lee-dohm]

:wave: Thanks for reaching out!

I'm not entirely sure what the ultimate goal is here, who is intended to benefit from it, and what that benefit is. Can you give us some more details?

[jugglinmike]

Sure! The goal is to use the tests in Test262 to verify that the lexer in this project is in-line with the latest version of ECMASCript. Specifically: that it correctly accepts input that includes only valid tokens and rejects input that cannot be tokenized.

This project would benefit from the increased coverage, potentially discovering and fixing issues in advance of error reports from users. With the right automation in place, the maintainers could easily update Test262 to receive the new tests which are continuously being added to that repository.

Test262 would also benefit. Following the integration I'm proposing, any new bugs reported against this project could be interpreted as coverage holes in Test262. Maintainers could then add the missing tests, and all the other consumers of the test suite would receive better test coverage.

[lee-dohm]

@maxbrunsfeld This sounds like it could be great for the tree-sitter JavaScript parsing. What do you think?

[maxbrunsfeld]

@jugglinmike Yeah, that'd be useful for tree-sitter-javascript. Currently as part of its CI, it parses npm and all of its dependencies, but Test262 would make a great addition. Like you said, we'd need to some how pull out the actual valid JavaScript code from the test files.

[jugglinmike]

This is my first time seeing tree-sitter-javascript. It appears to be a full-fledged parser. Is that right?

I ask because Test262 already supports consumption by parsers. Since tree-sitter-javascript has a Node.js build process, it could use test262-stream, a tool built for this purpose and currently used by Babel's parser, Esprima, and JSHint. No change to Test262 would be necessary.

The lexer use case is specifically interesting to me because it would require additional metadata in Test262. That said, the overhead of maintaining that metadata would only make sense if someone were using it. Would you folks be interested in testing this project in Test262 if it were well supported?

[maxbrunsfeld]

I don't personally see us investing in testing the TextMate grammar with test262, but I could see setting up tree-sitter-javascript to parse all that code on CI.

[jugglinmike]

Too bad--I really wanted an excuse to introduce lexing metadata into Test262! Anyway, here's a rough patch to integrate Test262 with tree-sitter-javascript: https://github.com/tree-sitter/tree-sitter-javascript/pull/83