yhirose / cpp-peglib

A single file C++ header-only PEG (Parsing Expression Grammars) library
MIT License
883 stars 112 forks source link

Cant seem to parse whitespace in precedence climbing macro #174

Closed stoneRdev closed 3 years ago

stoneRdev commented 3 years ago

So, when using macros to perform precedence climbing, %whitespace doesn't capture anything, and using precedence climbing seems to exclude other tokens. So parsing things like a=1 is fine, but when whitespace appears it goes bonk. Here's an example grammar explaining what I mean:

PROGRAM <- ASSIGNMENT
ASSIGNMENT_TOKEN <- ("=" !"=")
ASSIGNMENT_INFIX(BT,OP) <- BT (OP BT)* {
    precedence
        R =
}
ASSIGNMENT <- ASSIGNMENT_INFIX(LEFT_EXPR,ASSIGNMENT_TOKEN)
LEFT_EXPR <- [a-zA-Z0-9]+

Parsing the sample input a=1 is fine, but parsing a = 1 produces 1:2 syntax error, unexpected ' ', expecting <LEFT_EXPR>. Adding the %whitespace <- [ \t\r]* rule does not fix this, nor does making a ~WS <- [ \t\r]* and adding it to the macro like so:

...
ASSIGNMENT_INFIX(BT,OP) <- BT (WS OP WS BT)* {
    precedence
        R =
}
...

Infact, as soon as another token gets added to this maco, it complains that 3:1 'precedence' instruction cannot be applied to 'ASSIGNMENT_INFIX'.

So how is one expected to parse whitespace in the precedence climbing macros?

I will try to figure it out but any help is greatly appreciated

yhirose commented 3 years ago

@stoneRdev, %whitespace only works on tokens. So you have to apply token operators < ... > to token rules. Here is the example:

ASSIGNMENT_TOKEN <- < ("=" !"=") >
LEFT_EXPR <- < [a-zA-Z0-9]+ >
%whitespace <- [ \t\r]*

Hope it helps!

stoneRdev commented 3 years ago

Yes that was the ticket! Much appreciated! And thank you for your work on this project!