Re-write the existing DFA implementation as a CDFA, which is a decoration of a standard DFA, which supports default transitions natively, as well as partial-states.
Partial States: child states of another state which represent a single theoretical sequence of characters to transition from one state to another. For a partial transition (transition through partial states) to complete, it must be entirely matched, otherwise the CDFA will backtrack to the parent.
e.g. if the sequence 'abcd' transitions from state START to ABCD, this can be represented in a CDFA with partial states START -> START_A -> START_AB -> START_ABC -> ABCD.
Since partial states have a parent state, they maintain the default transitions (and any other properties) of their parent.
e.g. if the CDFA in the previous example encountered "abcf", the transition from START_ABC would follow the default matcher of START, without removing "abc" from the input stream.
CDFAs will provide a more performant and long-term solution to multi-character lexing.
srhickma
commented
6 years ago
Re-write the existing DFA implementation as a CDFA, which is a decoration of a standard DFA, which supports default transitions natively, as well as partial-states.
Partial States: child states of another state which represent a single theoretical sequence of characters to transition from one state to another. For a partial transition (transition through partial states) to complete, it must be entirely matched, otherwise the CDFA will backtrack to the parent. e.g. if the sequence 'abcd' transitions from state START to ABCD, this can be represented in a CDFA with partial states START -> START_A -> START_AB -> START_ABC -> ABCD.
Since partial states have a parent state, they maintain the default transitions (and any other properties) of their parent. e.g. if the CDFA in the previous example encountered "abcf", the transition from START_ABC would follow the default matcher of START, without removing "abc" from the input stream.
CDFAs will provide a more performant and long-term solution to multi-character lexing.