This PR is kinda a success and a failure: All of the constants are made, but the lexing of operators doesn't work. So,
I will remake the lexer, it will be a stack based lexer instead of an iterator based lexer. It will be simpler to work with.
The make_tokens function will be separated in two functions, make_token that lexes the input for one token and return it; if there is a comment the function will lexe and return None and the function lexe that lexes the input (with make_token function) while end of file is reached
This PR is kinda a success and a failure: All of the constants are made, but the lexing of operators doesn't work. So, I will remake the lexer, it will be a stack based lexer instead of an iterator based lexer. It will be simpler to work with. The
make_tokensfunction will be separated in two functions,make_tokenthat lexes the input for one token and return it; if there is a comment the function will lexe and return None and the functionlexethat lexes the input (withmake_tokenfunction) while end of file is reached