Tokenize Start/End, Whitespace, Integers, and Punctuators - Githubissues

chharvey / counterpoint

A robust programming language.

GNU Affero General Public License v3.0

2 stars 0 forks source link

Tokenize Start/End, Whitespace, Integers, and Punctuators #1

Closed chharvey closed 4 years ago

chharvey commented 4 years ago

Tokenize a program into different types of tokens.

FileBound ::= #x02 | #x03

Whitespace ::= [#x20#x09#x0A#x0D]+

IntegerLiteral ::= [0-9]+

Punctuator ::= "+" | "-" | "*" | "/" | "^" | "(" | ")"

Tokens for \u0002 and \u0003 represent start-of-text and end-of-text respectively.
Whitespace tokens contain one or more of the following: spaces (\u0020), tabs (\u0009), line feeds (\u000a), and carriage returns (\u000d).
Integer literal tokens contain one or more base-10 digits 0-9.
Each punctuator token contains one of the symbols shown above.

chharvey commented 4 years ago

commits 34b5cb4 , ffee94c , cb034ce close this