espree.tokenize doesnt tokenize comments

[ChrisMGeo]

I'm trying to create a syntax highlighting tool and am using espree which is fantastic, as it's easy to use in just 4 lines. However I don't know if this is due to my lack of research or just bad knowledge in JS, however the espree.tokenize function doesn't return comments in the following code:

const espree = require("espree");
var tokenized = espree.tokenize("var i = 0;\nfunction sayName(name){\n\tconsole.log('Hi '+ name)\n}\n//omg this is a comment\nconst espree=require('espree')", {comment:false});
console.log(tokenized);

Is there a way where I can show the comments as well and could I remove any token ignorance.

[ChrisMGeo]

I have also tried comment: true; as well but I don't even know if that is related to this or not.

[nzakas]

This is working as intended. Comments are not returned as tokens (this mimics the behavior of Esprima's tokenize() method). If you specify comments: true, then there is an extra comments array on the returned value:

const tokens = espree.tokenize(code, { comments: true });
tokens.comments.forEach(/* whatever */);

You can then use the locations of the tokens and comments to merge them into one array if you'd like.

[kaicataldo]

Closing this issue as it looks like the question has been answered. Please feel free to visit us in the ESLint Gitter if you have any other issues!