Editing JSON while preserving formatting (and comments)

[johnspackman]

Hi @vtrushin

I have a fork of json-to-ast which I've extended to firstly support comments and secondly to allow the JSON to be modified and written back out again while preserving comments and all whitespace. This is really useful for us (we are the Qooxdoo framework - https://github.com/qooxdoo) because it allows .json configuration files to be tweaked on the fly by code (eg we're developing a command line tool that will work similar to npm install module).

Is this something that you would be interested in receiving as a pull request? You can see the work to date at my personal repo here: https://github.com/johnspackman/json-to-ast/tree/editable-json

The changes to date are principally:

• Tokeniser is modified to collect whitespace and comments; instead of returning a tokenList array, an instance of a class Tokenizer is used which can selectively skip the whitespace and comments
• Parser is modified to use Tokenizer instance, add startToken and endToken properties that record where the AST node is in the token stream, and add leadingComments and trailingComments
• Add src/stringify.js that adds a reprint method to output a POJO object, using the whitespace and comments found in the original AST. Also included is a prettyPrint method that pretty prints from an AST tree
• Gulp process was modified to support source maps and as part of this the source files are compiled separately instead of into one parser.js; the new main.js retains backwards compatibility with the existing json-to-ast API.

At the moment, the code is functional and passing units tests and we're about to start testing in anger as we add it to our tools; there's no documentation (not that this really requires much), and this should be 100% backwards compatible with your existing release.

If you would like to incorporate this I would be willing to make changes to meet your requirements, eg coding standards, documentation, etc; alternatively my thoughts are that I would either incorporate it into our tool as is, or possibly release it as another npm module under another name (with full credits to you of course)

Regards John

[vtrushin]

Hi @johnspackman Sorry for my long answer.

It flatters me that you use it in your framework

Tokeniser is modified to collect whitespace and comments; instead of returning a tokenList array, an instance of a class Tokenizer is used which can selectively skip the whitespace and comments That good decision!

I'll take a closer look at your code today or tomorrow.

I see you use comments, which are not JSON-compatible. Why not use YAML instead? It's more config-friendly format :-)

[johnspackman]

Hi Vlad,

Great, I'm really glad you're interested! I've pushed a few fixes since my first post so it's a bit more battle tested now too 😄

We looked at YAML but decided not to use it because we wanted to keep the config as Javascript-like as possible, and while you can use JSON it's a superset that adds very a un-JSON style. We also wanted an API to make changes to the configuration file and preserve the user's comments & layout so a custom JSON parser like your json-to-ast was a perfect fit.

Regards John

[buremba]

@johnspackman take a look at http://jsonnet.org. We're basically using this library to generate Jsonnet from JSON.

[johnspackman]

@buremba thanks, it looks interesting - is it possible to use jsonnet API to edit the jsonnet code? I don't mean rewriting the JSON, I want to edit it in place, preserving the author's original commenting and white space.

[buremba]

Yes, you can make modifications in place and render the output as JSON. @johnspackman

[johnspackman]

@buremba that's not quite what I meant - can I use an API to edit the original myfile.jsonnet, make changes programatically, and then write them back to myfile.jsonnet but preserving all the user's comments and formatting in the original file?

Converting myfile.jsonnet to myfile.json would be a separate step that would be done by the code that wants to read the configuration file.

[buremba]

Jsonnet supports comments and compiles to JSON. I don't know about the formatting though. The idea is that you maintain Jsonnet files which are more user-friendly and compile to JSON as a configuration file for your system.

[johnspackman]

i get that, but that is not the purpose of my fork and "editable json". The purpose is that it is easy for a human to edit a json source file and then for it to be parsed by code - but it is a lot trickier for a computer program to edit the json and preserve comments and formatting.

EG if you wrote code that reads and parses this JSON:

{
   "a": /* should be between 1 and 10 */ 5,
   "c": 6 //     This is SIX
}

and you want your code to edit that file and change "a" to 6, but leaving the comment where it is, and then write the file out again - you can't do it (without something like my editable json fork)