The string "1.5" tokenizes as 3 tokens, "1", ".", and "5".
This is a consequence of the regex for INTEGER being tried before FLOAT.
This has probably escaped attention because the output file will read "1.5" anyway, since there are no whitespace tokens separating them.
However, if some application is looking at the token output rather than a single consolidated string or file, it will get the wrong result. An example would be using pcpp as a front end to a C compiler.
A simple rearrangement of the rules in parser.py, and making them both functions, fixes the problem.
ned14
commented
9 months ago
The string "1.5" tokenizes as 3 tokens, "1", ".", and "5".
This is a consequence of the regex for
INTEGERbeing tried beforeFLOAT.This has probably escaped attention because the output file will read "1.5" anyway, since there are no whitespace tokens separating them.
However, if some application is looking at the token output rather than a single consolidated string or file, it will get the wrong result. An example would be using pcpp as a front end to a C compiler.
A simple rearrangement of the rules in
parser.py, and making them both functions, fixes the problem.