I tried your tokenizer and I have problems with #if and #ifdef in C programming language. I think the problem comes with your handling in lines 84 to 91 from your cpp_tokenizer.py file.
You decide that all i's after a # are considered include. But that can also be an if, ifdef or ifndef.
I tried your tokenizer and I have problems with #if and #ifdef in C programming language. I think the problem comes with your handling in lines 84 to 91 from your cpp_tokenizer.py file. You decide that all i's after a # are considered include. But that can also be an if, ifdef or ifndef.
I hope you can fix this.