Currently, each token's data is held in the TokenData enum, which includes values of literals:
// From lex/token.rs
#[derive(Debug, Clone, PartialEq)]
pub enum TokenData {
/// Token is a numeric literal
NumberLiteral(f64),
/// Token is unit type literal `()`
UnitLiteral,
/// Token is boolean literal `true` or `false`
BoolLiteral(bool),
// Ident, Symbol, Keyword, TypeName, ...
}
This literal information should not be stored on tokens themselves. It's not used this early in the compiler pipeline, and should only exist on the LiteralValue AST type. This is actually duplicated in the Literal struct:
// From ast/expression.rs
#[derive(Debug, PartialEq, Clone)]
pub enum LiteralValue {
/// Literals `true` and `false`
Bool(bool),
/// Numeric literals
Float(f64),
/// `()`
Unit
}
/// Represents a literal expression, such as a boolean or number.
#[derive(Debug, PartialEq, Clone)]
pub struct Literal {
token: Token,
value: LiteralValue
}
This value is given in Literal::new(Token, LiteralValue) by the LiteralParser (see parse/parsers/expression/literal.rs).
This data should not live on the Token object as it should not exist that early in the pipeline, nor should it be duplicated. TokenData should be removed in favor of TokenType and literals should be parsed in LiteralParser.
Currently, each token's data is held in the
TokenData
enum, which includes values of literals:This literal information should not be stored on tokens themselves. It's not used this early in the compiler pipeline, and should only exist on the
LiteralValue
AST type. This is actually duplicated in theLiteral
struct:This value is given in
Literal::new(Token, LiteralValue)
by theLiteralParser
(seeparse/parsers/expression/literal.rs
).This data should not live on the
Token
object as it should not exist that early in the pipeline, nor should it be duplicated.TokenData
should be removed in favor ofTokenType
and literals should be parsed inLiteralParser
.