[WIP] Proposal : Change ANSI to Token parsing from regex to state machine

This pull request is a proposal showcasing a functional implementation of ANSI parsing using a state machine instead of regex

Motivation

Up until now the parsing of ANSI to Token representation was done using regex

https://github.com/bczsalba/pytermgui/blob/70f2f580256bc88037434319cc90b69041f545e2/pytermgui/regex.py#L7-L9

Basically the regex are matched against the ANSI and if it trigger a Token is emitted. Problem is that it easily cause issue for corner cases. Last one was that hyperlink closing sequence OSC 8;; ESC \ needed to be paired with an hyperlink opening sequence, causing un-opened hyperlink closing sequence to be written in plain as a result.

The behavior documented here, used as a reference for pytermgui behavior, specify that hyperlink closing sequence should always be sent to the terminal

As such, in terminal emulators an OSC 8 escape sequence just changes the hyperlink (or lack thereof) to the new value. It is perfectly legal to switch from one hyperlink to another without explicitly closing the first one. It is also perfectly legal to close a hyperlink when it's not actually open (e.g. to make sure to clean up after a potentially unclean exit of an application).

So changes had to be made to follow this specification.

This change could be made using regex but would quickly become hairy with the need to save the context somewhere, regex becoming very big and unreadable, etc...

I think that for implementing future feature, and for readability, it's better to use a state machine and a character per character reading of the ANSI.

Implementation

Showcased implementation is not as cleaned as it could be but I didn't want to go too deep while I'm not sure this change is wanted.

The state are defined like this:

ESC="\x1b"

CSI="["
SGR="m"
CURSOR="H"

OSC="]"
HYPERLINK="8"

ST="\\"

SEP=";"

You can notice I differentiate CSI and OSC commands. I think it will help implementation of new feature to have this granularity at the parsing level.

The state machine implementation is in tokenize_ansi. Some cases are not taken into account yet

For example what happens if after a CSI is never terminated by a known character ?
What if and OSC if followed by an unknow command ?
What error should we throw and when to throw them ?

Stability

All the test are currently passing.

bczsalba / pytermgui