This moves code common to all lexers to a runtime library called lexgen_util.
This library is not to be used by users.
This has a few advantages:
Code generator becomes simpler and generates less code
It becomes possible to define multiple lexers in the same scope
We avoid generating same types again for things like locations in the input.
In addition to redundancy, this causes problems with nominal typing (i.e.
Location types for different lexers become distinct).
This also implement location tracking. Instead of byte indices, we now return
struct Loc {
line: u32,
col: u32,
byte_idx: usize,
}
This moves code common to all lexers to a runtime library called
lexgen_util
. This library is not to be used by users.This has a few advantages:
Code generator becomes simpler and generates less code
It becomes possible to define multiple lexers in the same scope
We avoid generating same types again for things like locations in the input. In addition to redundancy, this causes problems with nominal typing (i.e.
Location
types for different lexers become distinct).This also implement location tracking. Instead of byte indices, we now return
Fixes #26