search

Servostar / gemstone

Gemstone programming language compiler (GSC)
GNU General Public License v2.0
3 stars 0 forks source link

5 create tokenizer #15

Closed flixm16 closed 6 months ago

flixm16 commented 6 months ago

Created all tokens for symbols and keywords used in the base language.

keyword token starts with "Key" and ends with the Keyword itself. Example: "KeyInt" for "int" Value token starts with "Val" Operator token starts with "Op" Compiler function token starts with "Fun"

single symbols don't have a token and can be used with with single quotes

Every Keyword was successful recognized

Servostar commented 6 months ago

Operators either passed-through as character or token. Should be uniform -> pass Operator always as token

Servostar commented 6 months ago

Missing floating point literals, got integer and string