Support multiple states when parsing rules

[tancnle]

Prior to this change, we can only push one state per rule. Many lexers need multiple states to work as expected. For example, some languages allow multiline comments to be nested.

For example, if we want to match a comment containing a directive, something like:

/* <processing directive>    rest of comment */

This also allows us to have parity with Pygments support of multiple states .