Sync lexer punctuations with ZetaSQL

[apstndb]

Some punctuations are not yet handled in memefish.Lexer. It cause panic when lexing even if parsing is not needed.

I think it is safe to add all punctuations in ZetaSQL.

https://github.com/google/zetasql/blob/194cd32b5d766d60e3ca442651d792c7fe54ea74/zetasql/parser/flex_tokenizer.l#L513-L552

• [x] (
• [x] [
• [x] {
• [x] )
• [x] ]
• [x] }
• [x] *
• [x] ,
• [x] =
• [ ] += KW_ADD_ASSIGN
  • It seems to be used in options_assignment_operator
• [ ] -= KW_SUB_ASSIGN
  • It seems to be used in options_assignment_operator
• [x] !=
• [x] <=
• [x] <<
• [x] =>
• [ ] -> KW_LAMBDA_ARROW

  • 153

• [x] <
• [x] >
• [x] >=
• [x] ||
• [x] |
• [x] ^
• [x] &
• [x] +
• [x] -
• [x] /
• [x] ~
• [ ] ?
  • Positional parameter is available in BigQuery, but Cloud Spanner has not yet supported.

    Positional parameters are not supported

• [ ] !
  • It seems not to be used.
• [ ] %
  • GQL #99
• [ ] |> KW_PIPE
  • Pipe syntax
• [x] @
• [ ] @@ KW_DOUBLE_AT
  • System variables in BigQuery
• [x] .
• [ ] :
  • NEW constructors #119
  • GQL #99
• [ ] \ BACKSLASH
  • Only for lenient macro expansion
• [x] ;
• [ ] ${unquoted_identifier} MACRO_INVOCATION
  • internal names in ZetaSQL?
• [ ] ${decimal_digits} MACRO_ARGUMENT_REFERENCE
• [ ] $
  • It seems not used.

Except MACRO_INVOCATION and MACRO_ARGUMENT_REFERENCE would be better to be implemented.

This makes Lexer more robust even to new lexical structures that are being tested privately.