daeisbae / AParser

Yet another programming language for educational purpose and simplicity
https://daeisbae.github.io/AParser/
MIT License
1 stars 0 forks source link

String Lexing Fix (Do not lex '\"' as end of the string) #20

Closed daeisbae closed 6 months ago

daeisbae commented 6 months ago

Previously when lexing the string "Hello \" World", it will return

>>> "Hello \" World"
Token( TokenType: String Value: 'Hello \" World' )
Token( TokenType: Whitespace Value: ' ' )
Token( TokenType: Identifier Value: 'World' )
Token( TokenType: String Value: '' )
Token( TokenType: End of line Value: '' )

Which is not a expected result. The char '\"' should be converted to double quote inside the string NOT being lexed individually.

Now it is improved to

>>> "Hello \" World"
Token( TokenType: String Value: 'Hello " World' )
Token( TokenType: End of line Value: '' )

Besides the change in string function inside the lexer class, lexing string and null test code is included