Forms like 'ab'c are rejected, so we need some way to explain why they don't tokenise as two consecutive LIFETIME_OR_LABEL tokens.
I think the best way to do this, given the Reference's current approach, is simply to add "not immediately followed by '" to the lexer rules for the lifetime-like tokens.
That matches what the implementation (lifetime_or_char()) is doing, so it's not likely to be wrong, and this chapter already has some cases of lookahead of this sort.
It also means there can be no ambiguity between CHAR_LITERAL and these tokens (I think the intent is that we have a traditional "longest matching token wins" rule, which would give the right result here, but that isn't explicitly stated anywhere).
Forms like
'ab'care rejected, so we need some way to explain why they don't tokenise as two consecutive LIFETIME_OR_LABEL tokens.I think the best way to do this, given the Reference's current approach, is simply to add "not immediately followed by
'" to the lexer rules for the lifetime-like tokens.That matches what the implementation (
lifetime_or_char()) is doing, so it's not likely to be wrong, and this chapter already has some cases of lookahead of this sort.It also means there can be no ambiguity between CHAR_LITERAL and these tokens (I think the intent is that we have a traditional "longest matching token wins" rule, which would give the right result here, but that isn't explicitly stated anywhere).