search

ianh / owl

A parser generator for visibly pushdown languages.
MIT License
746 stars 21 forks source link

hex integer #30

Closed ozbolt-abrantix closed 5 months ago

ozbolt-abrantix commented 2 years ago

If I try to implement another "built-in token class" like hexinteger (I will follow the implementation for other builtins), would the PR be excepted?

Unfortunately for my usecase I can not use 0xABC integer matcher with 0x.

ianh commented 2 years ago

Sure, that sounds fine -- to be clear, the token would match hex integers like ABC rather than 0xABC, right? I'd also say the name of the token class should be hex-integer rather than hexinteger to match other hyphenated built-ins like .line-comment-token.

ianh commented 2 years ago

Though, on second thought, there's an issue of which token type has priority between identifier, integer, and hex-integer, since 3 could be either decimal or hexadecimal and A could be either an identifier or a hex digit. It might be better to implement this as a custom token using the .token keyword in the grammar.

ianh commented 5 months ago

Closing this; given the ambiguities between the different token types, I think this is something that should be done using a custom tokenizer (or, if more advanced lexing is ever added to Owl, done using whatever mechanism is added for that).