Compiler to Json package (@toml-tools/toml-to-json)

[bd82]

• [x] Evaluate performance advantages of not using the CST for this.
• [ ] Evaluate performance advantages of using custom token patterns, could those be used to avoid post-processing? Possible use cases:
  • when lexing a date could we create the JS Date Object on the fly?
  • processing backslash line terminator in multiline strings
  • stripping strings delimiters
  • Numerical values conversion to Number during lexing.
• [ ] Evaluate additional "fastest" API relevance (e.g only offset position tracking).

[bd82]

Evaluate performance advantages of not using the CST for this.

Parsing seems 50-70% faster when not building the CST. So its likely ~50% performance boost could be gained by implementing a variant without CST creation.

The cost would be duplicating ~200 LOC (the parser itself) to insert the JSON building code into the parser flow. That duplication's alignment could be tested automatically by comparing the grammar AST structure. In addition the tests of the parser could be extracted to a separate dev package in this mono repo so we will avoid any duplication in the tests.