Add option to extend the beautifier with custom tokenizers/formatters

[bitwiseman]

Several recent issues filed would require siginifcant changes to the core tokenizing, parsing, and/or formatting code. For example, #234, and #223. The user implemented #234 and we accepted it because it was basically self-contained, but #223 would require quite a bit more work and doesn't seem to be part of any javascript spec.

Still, there are many flavors of javascript and these javascript-ish dialects are proliferating. If we added a few well-defined extension/plugin points we could better focus on our core goals. Over time we could even add some plugins to the distribution, providing a greater value to users.

Same principles would apply, extensions would need to not break the core features (and possibly other plugins), and would need to have tests.

[evocateur]

I definitely agree with the spirit of this proposal. Keep the core focused, allow powerful plugins.

The implementation, however, seems to beg for an AST (#200) that could be sent to plugins at well-defined steps (before/after default transforms, before "printing", etc). The various options we have now, like brace-style, could even be implemented as built-in plugins (similar to Express middleware, in my mind (perhaps addled by all the middleware-related work I've been doing recently)).

[bitwiseman]

Yeah, that's about where I was going, though I was hoping we could avoid a full AST.

[prettydiff]

If you would like you could use the Pretty Diff parsers and maintain independent beautification. I would have some work to do on my end to make this possible. Currently the parser, beautifier, analyzer, and other features are in the very same file for a respective language category. I would need to break these apart so that the parser is in a separate file for easier consumption. I would also need to write additional documentation to explain the API and some qualities that right now are purely internal or for things that I have not explicitly thought about explaining.

If you are interested just let me know I will open an issue to make this happen.

Some details about the parsers:

• They produce parallel arrays instead of ASTs. Think of this as outputting data in a table instead of like a JSON object.
  • The advantage is that data is faster to access by a respective array instead of walking the tree for a particular token
  • If you want to extend the tool for advanced analysis you can reorder the data by sorting on any given array and using a custom written sort function to ensure the indexes of each parallel array are likewise sorted. I was going to write some really advanced accessibility analysis using the markup parser back in December, but the work was blocked pending other enhancements. Those blocks are gone, but I haven't gone back to do that work as I have been recently been distracted with a new software project and work/life issues.
• The parsers are fairly robust and complete.
  • They are written in a way to never have collisions with extended unicode support
  • They support a variety of syntax, such as the markup parser supporting more than 20 delimiter types for a variety of template schemes
  • Each of the three parsers are several years old now and matured to the point where defects are pretty rare
• Format interchange, all of the parsers can be enhanced, provided the need should arise, to interchange tokens between them.
  • The markup parser has always used the JavaScript and CSS parser as such code can be embedded within markup code
  • JSX code meant a requirement for the JavaScript parser to send code segments to the markup parser. This interchange is recursive as JSX markup tokens could contain JavaScript code in the form of escapes, comments, and so forth
  • The JavaScript parser does not send anything to the CSS parser and the CSS parser does not currently send anything to the markup or JavaScript parser, but if the need should arise this wouldn't be a challenge.
• Structure self-awareness
  • The JavaScript parser's parallel arrays include the arrays: depth and begin
  • The depth array describes the code structure this token resides in (context)
  • The begin array provides the token index where this structure starts
  • You can simulate or form an AST using those two arrays. I have never bothered to do this, because I have always thought an AST would be slower to access and contain a lot of unnecessary decoration