JSON parsing/handling

[madmongo1]

Hi Vinnie, I saw your post in slack regarding boostified JSON parsing.

I felt it prudent to you let you know some of the use cases I have come across when using JSON in a highly available, scalable data distribution system written in c++.

1. Sometimes you don't want to parse all of the JSON into an object hierarchy.

There are many scenarios in my system in which some parts of the JSON need to be tokenised and consulted in-place prior to forwarding the data to the correct subsystem. Sometimes that subsystem's job is to simply forward the data on to the client unmodified.

In my software stack I use both nlohmann json (for ease) and jsmn for tokenisation. Each have their limitations. RapidJSON is also being considered as a drop-in to improve performance.

To me this argues for separating concerns within a JSON library if it is to be truly multi-purpose. The concerns are:

a) tokenisation b) parsing c) serialisation d) structured storage (by reference, value, etc)

and perhaps

e) partial tokenisation (i.e. finish once you find the token you want)

2. Sometimes you want to mutate a JSON object but not change the order of elements in its string representation.

Often I am taking JSON from one system, parsing, updating and forwarding to another. Sometimes masquerading as an invisible man-in-the middle between legacy systems.

Different languages have different internal storage rules for JSON objects. For example, they may be stored internally as a hashmap or ordered map. The JSON spec does not specify element ordering.

It would be helpful to me if a JSON utility library had the ability to preserve ordering while benefiting from hashing.

This is useful for regression testing for example.

3. Sometimes you want speed at the expense of utility, sometimes (mostly) utility is a priority.

I think this reiterates point 1.

Thanks for your continued contribution to the c++ community. It is very much appreciated.

[vinniefalco]

I agree with all of that, and I think the current design of the JSON library in the lounge meets these requirements. json::basic_parser calls virtual members for the tokens and parts. json::serializer is a separate class. The json::object type preserves ordering, and you can splice elements from one place to another.