polygonplanet / Chiffon

A small ECMAScript parser, tokenizer and minifier written in JavaScript.
https://polygonplanet.github.io/Chiffon/demo/javascript-parser-demo.html
MIT License
55 stars 4 forks source link

not ESPRIMA compatible anymore #7

Open ghost opened 8 years ago

ghost commented 8 years ago

@polygonplanet Great work with this projet!! But it seems to be a lot of minor differences vs Esprima now. Example run this test: https://github.com/jquery/esprima/blob/master/test/api-tests.js#L900

This will break! And there are also various AST differences.

E.g. Esprima now have this at the end of the object: sourceType": "script"

Esprima also now supports ECMA 2017 by default. And this module only supports ES2015.

And also Esprima have a delegate option as the third argument for the tokenizer.

ghost commented 8 years ago

@polygonplanet Here are a few more differences

etc...

ghost commented 8 years ago

This one give totally different result https://github.com/jquery/esprima/blob/master/test/api-tests.js#L263

Your module:

{ type: 'Identifier', name: 'a' }

Esprima:

{ type: 'Identifier', name: 'foo' }

polygonplanet commented 8 years ago

Thanks for your report. Chiffon is not strictly compatible with Esprima. However, since Esprima is faithful to the specification, it is using for comparison in the test. I will update it according to the latest ECMA-262 and ESTree specification.

ghost commented 8 years ago

Great project this one!

I also noticed in Esprima you can handle comments, but not in your module. Plan to add it? And Esprima also have a tokens options to list out all tokens. Very handy!

And this Esprima test output a different result then your module: https://github.com/jquery/esprima/blob/master/test/api-tests.js#L622

polygonplanet commented 8 years ago

I also noticed in Esprima you can handle comments, but not in your module. Plan to add it?

What means comments? Comment of tokenize()' defined token type?

Esprima also have a tokens options to list out all tokens

This seems very useful, I will consider implementation.

ghost commented 8 years ago

Hi, @polygonplanet ! How it goes with the changes? Did you add support for latest ECMA features?

With comments I was thinking of the same thing as you see in this tests here: https://github.com/jquery/esprima/blob/master/test/api-tests.js#L175 https://github.com/jquery/esprima/blob/master/test/api-tests.js#L181 https://github.com/jquery/esprima/blob/master/test/api-tests.js#L187

Note the { comments: true } option. You can choose to parse with and without comments.

One optimization idea is to parse template literals character by character because you have to be aware of context (consider template literals inside template literals). And also be aware of unterminated interpolations.

I also noticed that await function() {} ... is not implemented yet.

And I don't know if a Tree walker is something to consider? Acorn has one.