Getting rid of C-preprocessing

[mmottl]

Using the C-preprocessor was an awful hack to share code between parsers for strings and bigstrings without serious loss of efficiency. It may be worthwhile considering a pure OCaml alternative now that Flambda has landed. I think it might already be able to optimize this case, i.e. inline character access, etc. The downside is that one needs to compile with Flambda support then, which may be a bad default assumption.

[ghost]

Actually I rewrote the parser as an automata a few weeks ago :) The feature is being reviewed, but it's big so it's taking a bit of time. The new code is already faster than the old parser with 4.02 and the API is more flexible regarding the medium:

val feed : State.t -> char -> Stack.t -> Stack.t
val feed_eoi : State.t -> Stack.t -> outcome

[mmottl]

Thanks, good to know, an automata is probably a neater approach. It's interesting that it would perform better. I would guess that there was too little inlining with all these mutually recursive functions, leading to too many function calls.

[hhugo]

The new parser @diml was talking about has finally been released internally. We should be able to close the issue after the next push to gh.

[trefis]

Done.