Token: Add operators and punctuators

[Hywan]

Address https://github.com/tagua-vm/parser/issues/5, #8.

Specification

https://github.com/php/php-langspec/blob/master/spec/19-grammar.md#operators-and-punctuators

Progression

• [x] [,
• [x] ],
• [x] (,
• [x] ),
• [x] {,
• [x] },
• [x] .,
• [x] ->,
• [x] ++,
• [x] --,
• [x] **,
• [x] *,
• [x] +,
• [x] -,
• [x] ~,
• [x] !,
• [x] $,
• [x] /,
• [x] %,
• [x] <<,
• [x] >>,
• [x] <,
• [x] >,
• [x] <=,
• [x] >=,
• [x] ==,
• [x] ===,
• [x] !=,
• [x] !==,
• [x] ^,
• [x] |,
• [x] &,
• [x] &&,
• [x] ||,
• [x] ?,
• [x] :,
• [x] ;,
• [x] =,
• [x] **=,
• [x] *=,
• [x] /=,
• [x] %=,
• [x] +=,
• [x] -=,
• [x] .=,
• [x] <<=,
• [x] >>=,
• [x] &=,
• [x] ^=,
• [x] |=,
• [x] =& (removed),
• [x] ,,
• [x] ??,
• [x] <=>,
• [x] ....

[Hywan]

I have removed PLUS and DOLLARS, which are redundant with ADD and VARIABLE. I have introduced the following private tokens: COLON and QUESTION_MARK (they are respectively used by FUNCTION_OUTPUT and TERNARY_ELSE, and by NULLABLE and TERNARY_THEN). I have added FUNCTION_OUTPUT (didn't find a better name…) and NULLABLE.

[Hywan]

I have rebased my work too. Ready for another review @jubianchi :smiley:!

[Hywan]

ping @jubianchi?

[jubianchi]

@Hywan good for me! Glab you chose to extends the token! macro, I think it will let us do some cool things!