snirk-lang / protosnirk

The beginnings of a programming language
MIT License
1 stars 1 forks source link

Move literal data out of `TokenData` #109

Closed SnirkImmington closed 4 years ago

SnirkImmington commented 4 years ago

Currently, each token's data is held in the TokenData enum, which includes values of literals:

// From lex/token.rs

#[derive(Debug, Clone, PartialEq)]
pub enum TokenData {
    /// Token is a numeric literal
    NumberLiteral(f64),
    /// Token is unit type literal `()`
    UnitLiteral,
    /// Token is boolean literal `true` or `false`
    BoolLiteral(bool),

    // Ident, Symbol, Keyword, TypeName, ...
}

This literal information should not be stored on tokens themselves. It's not used this early in the compiler pipeline, and should only exist on the LiteralValue AST type. This is actually duplicated in the Literal struct:

// From ast/expression.rs

#[derive(Debug, PartialEq, Clone)]
pub enum LiteralValue {
    /// Literals `true` and `false`
    Bool(bool),
    /// Numeric literals
    Float(f64),
    /// `()`
    Unit
}

/// Represents a literal expression, such as a boolean or number.
#[derive(Debug, PartialEq, Clone)]
pub struct Literal {
    token: Token,
    value: LiteralValue
}

This value is given in Literal::new(Token, LiteralValue) by the LiteralParser (see parse/parsers/expression/literal.rs).

This data should not live on the Token object as it should not exist that early in the pipeline, nor should it be duplicated. TokenData should be removed in favor of TokenType and literals should be parsed in LiteralParser.

SnirkImmington commented 4 years ago

Closed in #110.