As discussed in #307 I decided to try to make it more readable by splitting the tokenization into one part and the reconstruction into strings in a separate part.
In contrast to the previous branch, this one rips out ocamllex and does the lexing manually, also reading as much as possible to avoid allocating too many objects.
To me this makes the code more understandable (especially since the list of tokens can be just inspected), but I'm happy to hear your opinion.
As discussed in #307 I decided to try to make it more readable by splitting the tokenization into one part and the reconstruction into strings in a separate part.
In contrast to the previous branch, this one rips out ocamllex and does the lexing manually, also reading as much as possible to avoid allocating too many objects.
To me this makes the code more understandable (especially since the list of tokens can be just inspected), but I'm happy to hear your opinion.