search

bzick / tokenizer

Tokenizer (lexer) for golang
MIT License
92 stars 6 forks source link

Parsing comments #21

Open ddorstijn opened 2 days ago

ddorstijn commented 2 days ago

It is currently possible not possible to parse comments. A work around might be for block comments to parse it as a string token. But inline comments need to know the end of the line, which is not possible as far as I am aware

bzick commented 2 days ago

Can you provide an example what you want to do?

ddorstijn commented 7 hours ago

I think every language with an inline comment could be an example. Take SQL:

select 1 + 1 -- This returns 2
from xyz

Javascript

const x = a + b; // This is an inline comment
myFunc();

Block comments would be something like this:

/* This query returns the result of 1 + 1
    Written by xyz
*/
select 1 + 1

And javascript

/* Return a + b */ return a + b;