Finish first pass of lexer

[hawkins]

• [x] Come up with a list of token types
• [x] Create token struct
  • type, text, length
• [ ] Tokenize input stream
  • From file? From stdin?
• [ ] Write output stream
  • To file? To Stdout? Where?

[hawkins]

Made more progress on lexing trivial forms of each token we support. Noticed we don't support -- / ++ inc/decrementing. Will want to add that eventually, but maybe not quite yet. A simple grammar always helps a first parser pass so we can add things like that in after the fact. We're already pretty complicated, so it's sort of a moot point, but I think it's about time for a feature freeze on first grammar.

Next steps:

• [ ] Testing more thoroughly, particularly the 'unhappy path' for tokens (i.e., unterminated string, unexpected eof, floats without right hand side numbers, shit like that)
• [x] Clean up code by merging _TEST #ifdef's only being in one spot
• [ ] Adding a _DEBUG flag for printf output to print more info throughout NextToken for debugging test fails
• [ ] Decide on a strategy for piping between PP -> LEXER -> PARSER -> COMPILER (i.e., pointer addressing sharing memory since its one c program? stdout piping? file piping?)
  • may 2020: I'm leaning toward pointer addressing for simplicity sake. creating files to pass is junky, and its only one program, so stdout doesn't make sense.
• Test segfaults on my Kali on Windows setup, but passes on my Mac. That needs to be triaged.

[hawkins]

I think this is closed by #5

lol no that prompted this issue, nvm