Closed daeisbae closed 6 months ago
Previously when lexing the string "Hello \" World", it will return
>>> "Hello \" World" Token( TokenType: String Value: 'Hello \" World' ) Token( TokenType: Whitespace Value: ' ' ) Token( TokenType: Identifier Value: 'World' ) Token( TokenType: String Value: '' ) Token( TokenType: End of line Value: '' )
Which is not a expected result. The char '\"' should be converted to double quote inside the string NOT being lexed individually.
Now it is improved to
>>> "Hello \" World" Token( TokenType: String Value: 'Hello " World' ) Token( TokenType: End of line Value: '' )
Besides the change in string function inside the lexer class, lexing string and null test code is included
Previously when lexing the string "Hello \" World", it will return
Which is not a expected result. The char '\"' should be converted to double quote inside the string NOT being lexed individually.
Now it is improved to
Besides the change in string function inside the lexer class, lexing string and null test code is included