search

bablr-lang / bablr-vm

A VM for enforcing language rules on agAST trees
MIT License
40 stars 2 forks source link

Define token types? #15

Closed conartist6 closed 1 year ago

conartist6 commented 2 years ago

Currently any language can define its tokens any way it wants. I'd like there to be some force that pulls people back towards a standard way of defining which tokens are which so that we don't end up with a million different ways of doing things.

I could mandate the use of certain token names, but that seems like it would unnecessarily limit the usefulness of the tool. Most likely I will want to ensure that there are ample rewards for sticking to my way of categorizing tokens.

conartist6 commented 1 year ago

I think finally figured this out. I've added a bunch of token types with intrinsic meaning, e.g. SymbolDefinition and LeftPunctuator. I'm to covert these back to more generic types in my base parser, e.g. Identifier and Punctuator. Then I'll make plugins that redefine the output, and those plugins will implement protocols to indicate that they conform to interfaces which (should) allow certain functionality to be provided.