Develop the Lexer

[mkpro118]

The lexer will take the regex string and convert it into a list of tokens. Tokens can be characters, operators like * or |, and special symbols like ( and )

The terminals recognized by our lexer are

terminal ::  CHAR := Any printable ASCII character except the terminals below.
terminal ::  OR := |
terminal ::  STAR := *
terminal ::  PLUS := +
terminal ::  QUESTION := ?
terminal ::  LPAREN := (
terminal ::  RPAREN := )

• [x] #2
• [x] #3
• [x] #4
• [x] #17

[mkpro118]

Lexer has been developed, with the following PRs

• 22

• 26

• 43 which is another set of PRs

  • 30

  • 36

  • 37

  • 38

  • 41

  • 42

• 57