osa1 / lexgen

A fully-featured lexer generator, implemented as a proc macro
MIT License
63 stars 7 forks source link

Update resetting current match in rules without a RHS #64

Closed osa1 closed 1 year ago

osa1 commented 1 year ago

Currently a rule without a RHS (e.g. $$whitespace,) resets the current match in the Init rule set, but it does not reset it in other rule sets.

For example, the lexer below

rule Init {
    $$whitespace,
    ...
}

rule MyRule {
    $$whitespace,
    ...
}

would work as if it's desugared to:

rule Init {
    $$whitespace => |lexer| {
        lexer.reset_match();
        lexer.continue_()
    },
    ...
}

rule MyRule {
    $$whitespace => |lexer| {
        lexer.continue_()
    },
    ...
}

While this is convenient when skipping whitespace (which is generally only done in Init), it's a special case that we need to keep in mind when using rules without a RHS.

This PR removes this special case. A rule without a RHS now always resets the current match.

$$whitespace,

Desugared to:

$$whitespace => |lexer| {
    lexer.reset_match();
    lexer.continue_()
},

Regardless of the rule set.

This is breaking change. A rule without a RHS in a non-Init rule set now needs to explicitly call continue to work as before. Example:

// Old code, this used to NOT reset the current match, but it now
// resets the current match
rule MyRule {
    $$whitespace,
    ...
}

// New version that works as before:
rule MyRule {
    $$whitespace => |lexer| lexer.continue_(),
}

Fixes #12