Currently we use Token objects to store intermediate values. We use these throughout the library and it always bothered me but I could never think of a better word for its usage. I think I finally discovered the usage difference.
In the lexer, we return Token objects and the parser parses with them. These Tokens represent some text in the Uniform program which we are interpreting. I think that usage is fine, it is what the Token was originally intended for.
In the evaluator and throughout the semantic objects (Identifier, BlockVariable, etc.) we also return Tokens as intermediate values for an expression. The expression test equals "" will yield Token(true, constants.TYPE.BOOL) or Token(false, constants.TYPE.BOOL). However, this is misleading because such a Token would imply that the user had typed the text true or false as literal Boolean values.
I think it would be helpful to separate this usage and use a different class to represent intermediate values such as what the evaluator works with. Maybe just a Value class which has a value and type (perhaps calling the inner value val to avoid confusion?). This would not have line or column information, since it does not represent any text in the source code. That may make error messages a little more complicated but I think a distinction between Token and Value would be beneficial.
Currently we use
Tokenobjects to store intermediate values. We use these throughout the library and it always bothered me but I could never think of a better word for its usage. I think I finally discovered the usage difference.In the lexer, we return
Tokenobjects and the parser parses with them. TheseTokensrepresent some text in the Uniform program which we are interpreting. I think that usage is fine, it is what theTokenwas originally intended for.In the evaluator and throughout the semantic objects (
Identifier,BlockVariable, etc.) we also returnTokensas intermediate values for an expression. The expressiontest equals ""will yieldToken(true, constants.TYPE.BOOL)orToken(false, constants.TYPE.BOOL). However, this is misleading because such aTokenwould imply that the user had typed the texttrueorfalseas literal Boolean values.I think it would be helpful to separate this usage and use a different class to represent intermediate values such as what the evaluator works with. Maybe just a
Valueclass which has a value and type (perhaps calling the inner valuevalto avoid confusion?). This would not have line or column information, since it does not represent any text in the source code. That may make error messages a little more complicated but I think a distinction betweenTokenandValuewould be beneficial.