Open davesnx opened 4 months ago
Currently we use 2 lexers that live inside Css_lexer, and run separate test suites: Tokenizer_test and css_lexer_test.
We need to unify those, expose a single API to tokenize a string of CSS and treat errors as an exception (Css_lexer.Error) (since menhir needs it), run a single test suite.
After the unification of lexers, we have only one. We still have 2 separate parsers with 2 separate techniques (menhir and rules with combinators and the ppx).
Would be nice to join them into the same package. There's a few benefits of having them togeter:
There's a bit of work left to unify the lexers which is the API, we have from_string and tokenize where one gets a recursive token structure, and the other a list of tokens. Both contain locations, it's a matter to only use one of them
css_parser.mly parses stylesheets/selectors/declarations/properties and values
Parser.re currently parses (and type-checks) properties + values (there's code in Parser.re that might make it possible to parse stylesheets/selectors and everything but isn't used)
Parser.re uses
value.rec
ppx whichs generates the parser fns for each property, while css_parser.mly uses menhir. I'm more inclined on having a hand-made parser or push for menhir Incremental API in order to call the value.rec's parsers, but it's up to see if it's a good idea.The driver calls one and the other depending at which stage of the parsing happens, in order to have success with https://github.com/davesnx/styled-ppx/issues/429 it's necessary to have more control over the parsing phase.
Resources
Mix Lexer and Parser like https://github.com/mnxn/eon and take a look at Sedlexing from here: https://github.com/FStarLang/FStar/pull/2203/files