编译器之词法分析

[xxleyi]

词法分析之 tokenization:

词法分析之实现:

词法分析

• tokenization
  • Treat program as string, identify the substrings and label each one with its token class
  • \tif (i == j)\n\t\tz = 0;\n\telse\n\t\tz = 1;
  • Whitespace
    • \t
    • \n\t\t
    • \n\t
  • Keywords
    • if
    • else
  • Operator
    • ==
    • =
  • Numbers
    • 0
    • 1
  • 单例
    • (
    • )
    • ;

An implementation must do two things:

• Recognize substrings corresponding to tokens
  • the lexemes
• Identify the token class of each lexeme
  • token is a pair
  • token class
  • lexeme
• left-to-right scan => lookahead sometimes required