Replacing ArduinoJson in the library

[mikee47]

The object structure is working and we have type information for everything. That means the structure is fully described including key and value names so copying these to RAM is un-necessary and wasteful.

So next step is to implement our new Store and associated objects to take care of serialisation and deserialisation, with an appropriate in-memory data store.

Serialisation is fairly straightforward and we can accommodate default value behaviour (issue #6).

For deserialisation we can use JsonStreamingParser.

In-memory storage

To confirm, ConfigDB::Object instances are only references to the actual data, which is managed by the Store.

The type information describes the data layout so RAM usage can be minimal. The contents of a store must be fully de-serialised into RAM - though see below under 'partial updates' for a possible future extension.

Object storage can be allocated as a single fixed-size block. We want to avoid embedding pointers in the data as this improves safety. It also makes binary serialisation/deserialisation (pickling) much simpler; we'll probably want that at some point.

The JSON integer type shall default to int32_t storage type, with implicit minimum (INT32_MIN) and maximum (INT32_MAX) range attributes. The storage type can be changed by setting these appropriately. For example, adding "minimum": 0 would change the property type to UInt32. Setting "minimum": 0, "maximum": 255" would change to uint8_t, etc.

Generated type information will be updated to include the range for each value. This can also allow invalid values to be rejected or clipped/truncated as required.

Let's leave out double-precision floating point as that's normally only used for calculations. Internally we might even consider storing floats as strings, so the application decides whether to convert to float or double themselves.

Store shall keep all string data in a pool using CStringArray. String properties shall be stored as an index into that array. e.g. #4 is fourth string. #0 is reserved to indicate default value applies. Entries in the pool are never modified. New or modified values are appended. Pool gets reset if store is reloaded.

Arrays can be stored in a similar way to strings, using indices into an object pool. The pool will probably be implemented using a Vector or std::vector. Object pool #0 contains the main object.

dbgen additions

To support integer property typing, minimum / maximum values require processing and the appropriate type information generated. It might also contain pre-calculated values such as total structure size to simplify memory allocation.

JSON Schema describes the $ref keyword which might be useful for repeated structures.

Support for enums would be handy.

Not sure if there's a standard way to define a custom value type, such as an integer with a specific range, other than adding min/max to every instance.

Partial JSON updates

There is a possibility here for optimising updates to part of a JSON file, for example just one object:

• During de-serialisation, store only the section of store.json we want to work with
• The in-memory store is updated as usual
• At commit time, create output file store.json.new
• De-serialise store.json with JsonStreamingParser
• If our store contains a section, output that, otherwise pass the de-serialised data through
• After de-serialisation, if our store hasn't been output (i.e. it wasn't in the original JSON), then output it
• Rename store.json as store.json.old
• Rename store.json.new as store.json

This helps keeps RAM usage under control.

[pljakobs]

I'm getting an error on "element.isContainer()" - which I can't find in JsonStreamingParser (from your repo, branch devel) - is that a change not yet commited?

[mikee47]

I've added the missing commit now.

[pljakobs]

I was just looking through the code, this is becoming so very elegant! You have taken the idea way farther than I've ever thought possible.

[mikee47]

I've merged this into develop now.