Currently, the lexer parses a define directive as a TokenStream::Define which just contains a vector of DefineTokens for the macro signature part and a vector of normal Tokens for the body part.
This setup inherently has a number of invariants that are not expressed through the type system and have already caused some confusion after a long break on working on this project, (not a comprehensive list):
If ident_tokens is empty, so will body_tokens.
If ident_tokens has a length of 1, this is an object-like macro, otherwise it's a function-like macro.
If ident_tokens has a length of 1 or more, the first token will always be a DefineToken::Ident(_) variant.
If ident_tokens is empty but body_tokens isn't, that means the immediate first character was invalid.
This is unnecessarily ambigious and complicated. We should refactor this to more cleanly express the invariants, probably through a new enum type.
Currently, the lexer parses a define directive as a
TokenStream::Definewhich just contains a vector ofDefineTokens for the macro signature part and a vector of normalTokens for the body part.This setup inherently has a number of invariants that are not expressed through the type system and have already caused some confusion after a long break on working on this project, (not a comprehensive list):
ident_tokensis empty, so willbody_tokens.ident_tokenshas a length of 1, this is an object-like macro, otherwise it's a function-like macro.ident_tokenshas a length of 1 or more, the first token will always be aDefineToken::Ident(_)variant.ident_tokensis empty butbody_tokensisn't, that means the immediate first character was invalid.This is unnecessarily ambigious and complicated. We should refactor this to more cleanly express the invariants, probably through a new enum type.