search

shapesecurity / shift-parser-js

ECMAScript parser that produces a Shift format AST
http://shift-ast.org/parser.html
Apache License 2.0
248 stars 28 forks source link

Incorporating Test262 #362

Closed jugglinmike closed 5 years ago

jugglinmike commented 7 years ago

I recently extended the Test262 project to include additional metadata for "negative" tests. Specifically: tests that are expected to produce early errors and syntax errors can now be automatically distinguished for tests that are expected to produce runtime errors.

My goal was to make the tests consumable by JavaScript parsers. I've since incorporated the tests into the JSHint project, and it's been very useful for fixing bugs and preventing regressions. I'd like to help other Free and Open Source parsers do the same.

Would the maintainers be interested in including Test262 in the automated test suite for this project? If so, I can follow up with some more specific practical questions (e.g. submodule usage, etc.) and go from there.

bakkot commented 7 years ago

We actually recently included the test262 parser tests in our tests! But we might want to include the rest of test262, yes.

I've been debating adding some fraction of the main test262 tests to the parser tests - the negative tests especially. In that case, there would be little value in further testing against the main test262 set. On the other hand, that duplication may not be valuable.

jugglinmike commented 7 years ago

There's a lot of value in the non-negative tests, even for parsers. If you do incorporate Test262 into that project, I would strongly recommend including all tests.

I share your reluctance about the duplication. In your recent comment on that issue, you suggest performing the inclusion as a build step. That's certainly an improvement over outright copying, but it also introduces a gate for consumers looking to get the latest version of Test262.

jugglinmike commented 5 years ago

You folks might be interested in test262-stream, a tool we built to help Node.js parsers consume Test262. It's being used by JSHint. Acorn, Esprima, and Babel, so there's less duplicated effort.

Whatever the technical implementation, I'm glad Shift is using Test262. Congrats!

bakkot commented 5 years ago

@jugglinmike Thanks! I ended up just depending on the upstream test262-parser directly; the stream-based API in test262-stream seemed a little heavyweight compared to a basic walker, and I needed to do manual filesystem operations anyway to enforce that the expectations file didn't get stale.