Implement token type module and complete the token types

[am-cid]

Token types contain:

1. Lexeme
2. Token
3. Delim id
4. ErrorType

Example: TokenType.BWEAK

1. "bweak"
2. "RESERVED_WORD" or "RESERVED_WORD_BWEAK" or "BWEAK"
3. "end"
4. ErrorType.BWEAK

passing it to self._peek() would be like

self._peek(*TokenType.BWEAK)
self._peek(*TokenType.CHAN)
self._peek(*TokenType.CAP)

---

also, complete the token types once implemented

[am-cid]

Concerns:

1. This is not applicable for identifiers and literals so we'd still have to do it manually there. This can only get passed as an argument for self._peek() which is not how we check for identifiers and literals
2. If all TokenTypes have their own ErrorType, wouldn't it make sense to just make the error type built in and not an enum from another module?

[HolyShaq]

Response:

1. Yup. I don't think there's any problem with that, though. We can still assign a TokenType for ids and literals even if we won't use the peek method for them.
2. That would make a lot more sense. I will try and refactor that part of the code once I implement this.

[HolyShaq]

Would it make sense if this is implemented under the token module? I'm thinking of something like this.

[Gidsss]

Yeah, looks good and promising. I'm down. So if kunwari nagcall ako print(TokenTypes.BWEAK.lexeme) output niya is yung "bweak" right?

Would it make sense if this is implemented under the token module? I'm thinking of something like this.

[HolyShaq]

Yep yep. Though, I'm making it so that the lexeme is independent from the token type class. Pacheck nalang sa branch ko if it makes sense HAHAHAHAH

Commit 3 for my branch

[am-cid]

Response:

1. Yup. I don't think there's any problem with that, though. We can still assign a TokenType for ids and literals even if we won't use the peek method for them.

The problem with that is that this class takes in a lexeme argument and identifiers don't have a fixed lexeme

Would it make sense if this is implemented under the token module? I'm thinking of something like this.

use __init__ instead of a subclass if you're going to stick to using Enum

from enum import Enum

class AClass(Enum):
    SAMPLE_CONSTANT = ("Sample arg 1", "Sample arg 2")

    def __init__(self, arg1, arg2):
        self.arg1 = arg1
        self.arg2 = arg2

print(AClass.SAMPLE_CONSTANT.arg1) # prints "Sample arg 1"

[HolyShaq]

PR for this issue!!! #14