Closed mikee47 closed 2 months ago
I'm getting an error on "element.isContainer()" - which I can't find in JsonStreamingParser (from your repo, branch devel) - is that a change not yet commited?
I've added the missing commit now.
I was just looking through the code, this is becoming so very elegant! You have taken the idea way farther than I've ever thought possible.
I've merged this into develop now.
The object structure is working and we have type information for everything. That means the structure is fully described including key and value names so copying these to RAM is un-necessary and wasteful.
So next step is to implement our new Store and associated objects to take care of serialisation and deserialisation, with an appropriate in-memory data store.
Serialisation is fairly straightforward and we can accommodate default value behaviour (issue #6).
For deserialisation we can use
JsonStreamingParser
.In-memory storage
To confirm,
ConfigDB::Object
instances are only references to the actual data, which is managed by the Store.The type information describes the data layout so RAM usage can be minimal. The contents of a store must be fully de-serialised into RAM - though see below under 'partial updates' for a possible future extension.
Object storage can be allocated as a single fixed-size block. We want to avoid embedding pointers in the data as this improves safety. It also makes binary serialisation/deserialisation (pickling) much simpler; we'll probably want that at some point.
The JSON
integer
type shall default toint32_t
storage type, with implicitminimum
(INT32_MIN) andmaximum
(INT32_MAX) range attributes. The storage type can be changed by setting these appropriately. For example, adding"minimum": 0
would change the property type toUInt32
. Setting"minimum": 0, "maximum": 255"
would change touint8_t
, etc.Generated type information will be updated to include the range for each value. This can also allow invalid values to be rejected or clipped/truncated as required.
Let's leave out double-precision floating point as that's normally only used for calculations. Internally we might even consider storing floats as strings, so the application decides whether to convert to float or double themselves.
Store shall keep all string data in a pool using
CStringArray
. String properties shall be stored as an index into that array. e.g. #4 is fourth string. #0 is reserved to indicate default value applies. Entries in the pool are never modified. New or modified values are appended. Pool gets reset if store is reloaded.Arrays can be stored in a similar way to strings, using indices into an object pool. The pool will probably be implemented using a
Vector
orstd::vector
. Object pool #0 contains the main object.dbgen additions
To support integer property typing, minimum / maximum values require processing and the appropriate type information generated. It might also contain pre-calculated values such as total structure size to simplify memory allocation.
JSON Schema describes the
$ref
keyword which might be useful for repeated structures.Support for enums would be handy.
Not sure if there's a standard way to define a custom value type, such as an integer with a specific range, other than adding min/max to every instance.
Partial JSON updates
There is a possibility here for optimising updates to part of a JSON file, for example just one object:
store.json
we want to work withstore.json.new
store.json
withJsonStreamingParser
store.json
asstore.json.old
store.json.new
asstore.json
This helps keeps RAM usage under control.