search

MysteryBlokHed / databind

Expand the functionality of Minecraft Datapacks.
https://databind.rtfd.io/en/stable
GNU General Public License v3.0
4 stars 0 forks source link

Rewrite tokenizer #54

Closed MysteryBlokHed closed 3 years ago

MysteryBlokHed commented 3 years ago

Right now, the tokenize() function will look for a colon preceded by a space before attemting to turn something into a keyword, and anything else is put into a token called NonDatabind. There are some problems with this system. In the way it's implemented, it's impossible for a token to start with another (eg. you couldn't have :def and :deffunc, since it would find def and stop looking for new keywords). It's also impossible to make a token that doesn't begin with :, which is pretty unusual for any language.

If the tokenizer was rewritten to just treat anything separated by whitespace as a separate token, then there would be a lot more freedom as to what tokens could be named, etc. Any space-separated tokens that weren't recognized by Databind would just be assumed to be normal Minecraft commands and kept as-is. However, this is not an issue for this PR. The following:

func test
say hi
endfunc

Could be tokenized as something like: [Token::DefineFunc, Token::FuncName("test"), Token::NewLine, Token::NonDatabind("say"), Token::NonDatabind("hi"), Token::NewLine, Token::EndFunc].

MysteryBlokHed commented 3 years ago

With this change the assignment operator .= should be changed to :=, as it's already used to initialize variables in some languages. Changes to the language's syntax should be done in a separate PR after this issue is closed.