tokay-lang / tokay

Tokay is a programming language designed for ad-hoc parsing, inspired by awk.
https://tokay.dev
MIT License
240 stars 7 forks source link

Sequence peeking input is being ignored #64

Open phorward opened 2 years ago

phorward commented 2 years ago

Given this program

T_EOL : @{
    peek EOF
    peek ' '
}

'Hello' T_EOL _ print("World")

and the input Hello, the call to peek EOF is ignored and afterwards rejected as it doesn't consume any input, even if its valid. This bug is a play between peek and the Op::ForwardIfConsumed operation which doesn't recognize that input was consumed because peek consumed input but reset it afterwards.

There is a workaround for this problem by defining T_EOL this way:

T_EOL : @{
    peek EOF accept
    peek ' '
}

It can also be reproduced when the two sequence are flipped, so

T_EOL : @{
    peek ' '
    peek EOF
}

Doesn't work either, but

T_EOL : @{
    peek ' '  accept
    peek EOF
}

does. Latter one does also work because the result is not reset, and the parselet accepts next:

.T_EOL@6     000 Frame(0)
.T_EOL@6     001 Fuse(11)
.T_EOL@6     002 Frame(0)
.T_EOL@6     003 Frame(0)
.T_EOL@6     004 CallStatic(6)
.T_EOL@6     005 Reset
.T_EOL@6     006 Close
.T_EOL@6     007 Accept
.T_EOL@6     008 Collect(0, 5)
.T_EOL@6     009 Close
.T_EOL@6     010 ForwardIfConsumed(7)
.T_EOL@6     011 Reset
.T_EOL@6     012 Frame(0)
.T_EOL@6     013 Offset(Offset { offset: 40, row: 3, col: 10 })
.T_EOL@6     014 CallStatic(7)
.T_EOL@6     015 Reset
.T_EOL@6     016 Close
.T_EOL@6    >017 Close
.T_EOL@6    --- Stack ---
.T_EOL@6     000 Empty
.T_EOL@6     001 Range(1..6, None, 0)
.T_EOL@6     002 Empty
.T_EOL@6    --- Frames ---
.T_EOL@6     000 Frame { fuse: None, capture_start: 3, reader_start: Offset { offset: 6, row: 1, col: 12 } }
.T_EOL@6     001 Frame { fuse: Some(12), capture_start: 3, reader_start: Offset { offset: 6, row: 1, col: 12 } }
.T_EOL@6    ip = 17 state = Ok(Next)
.T_EOL@6    returns None
phorward commented 3 weeks ago

The problem still exists in Tokay v0.6.6, with this updated version

T_EOL : @{
    Peek<EOF>
    Peek<' '>
}

'Hello' T_EOL _ print("World")