Lexer - C99 Update

[CalebLItalien]

Details

Now that we're transitioning to C99, we need to update the lexer to accept C tokens exclusively. This will mean adding some tokens and removing others, as well as updating unit tests.

Issue Type

• [X] Feature
• [ ] Bugfix
• [ ] Documentation
• [X] Refactoring
• [ ] Other:

Requirements

The following will be the acceptable tokens, all of which need to be unit tested:

• break
• case
• char
• const
• continue
• do
• double
• else
• enum
• float
• for
• if
• int
• long
• return
• signed
• struct
• switch
• unsigned
• void
• while
• + (addition)
• - (subtraction, unary minus)
• * (multiplication, dereference)
• / (division)
• % (modulo)
• = (assignment)
• > (greater than)
• < (less than)
• >= (greater than or equal to)
• <= (less than or equal to)
• == (equality check)
• != (not equals)
• && (logical and)
• || (logical or)
• ! (logical not)
• & (bitwise and, address of)
• | (bitwise or)
• ^ (bitwise xor)
• ~ (bitwise not)
• ++ (increment)
• -- (decrement)
• -> (pointer to member)
• . (member access)
• , (comma)
• ; (semicolon)
• : (colon)
• ? (conditional true)
• ( (left parenthesis)
• ) (right parenthesis)
• { (left brace)
• } (right brace)
• [ (left bracket)
• ] (right bracket)
• <identifier> (user-defined names)
• <constant> (integer, float, character, or string literals)
• Single-line comments start with //
• Multi-line comments start with /* and end with */

Notes

For unit testing this issue, I'd like you to have one test for each token (that just tests that one token in isolation) and then at least a couple tests per token aiming for edge cases and sequences of tokens. I'd also like you to organize the testing into a format similar to token.rs. This may involve making breaking changes to SyntaxElement, so make sure to discuss the progress of this issue with whoever is handling the C99 SynaxElement update.

[kjwelch]

3 more tokens that might need to be added are *= (multiplication assignment), /= (division assignment), and %= (modulo assignment).

[CalebLItalien]

@kjwelch We're going to hold off on assignment operators until a later date. Feel free to open an issue for them we can hold off of for now.

[kjwelch]

I changed the Token for MINUS and NEGATIVE to be one token that is parsed differently. The token is called DASH