rbertoche
commented
4 years ago Moved here, as asked. I'll read up the discussions you linked later today
Open rbertoche opened 4 years ago
rbertoche
commented
4 years ago Moved here, as asked. I'll read up the discussions you linked later today
I would like to propose to contribute with a spec for optimized serialization for any kind of struct, class or object.
It's similar to the metadata node you already proposed.
If it seems off-topic, personally I think it matches nicely with BJData and UBJSON's compact form: People come to binary format usually not willing to waste space repeating the same strings over and over. When simple UBJ_OBJECT are used for this purpose, that's what happens.
In contrast to opaque index or ndarrays, this spec allows for much better readability of the data and guarantees correct interpretation of data in the future by preventing you to lose record of which field is what or where. This is specially useful when data types and fields changes too often. I'm guessing you already know that.
In two cases I'm proposing new tags to UBJSON, to allow nesting inside those type's values, but the same could be done with reserved strings as you proposed, without changing or depending on UBJSON.
I think this spec would be specially useful when combined with reflection features that could allow automatic serialization and deserialization.
Inheritance could also be supported either with simple Single Table Inheritance or more sophisticated means.
I've turned a similar scheme of object serialization with UBJSON into a draft for this spec idea. See if you think that belongs somewhere in your project.
This idea is about an UBJSON container that holds both metadata and the object data to be stored in a compact way;
The metadata part is another container which contains values that specify the fields of all object types that will be stored later on the file.
Example of metadata:
Then each instance is represented in the data portion as an array of the appropriate type: UBJ_MIXED or something else or even ndarrays if it's scalar data.
The type of each data instance is identified either by a preceding integer "tag" in the case of an heterogeneous array, or in the case of homogeneous representation, by it's location on the file.
If the array index or integer tag matches the index of a metadata type entry, then that type entry describes the fields of that data entry.
Simple example of heterogeneous data:
In another example using heterogeneous data, object nesting is achieved using a new UBJSON tag "t". This tag would be followed by the integer tag or index of the class or type, then by it's contents.
UBJSON numeric type tags are omitted for readability:
An "Appointment" entry would look like this:
Example of linear homogeneous data in UBJ_MIXED arrays:
Another way to a similar nesting representation as above but using index-based reference, and other new UBJSON tag, this time "R" for reference.
This representation is useful when there are many repetitions.
Each R tag will be followed by a type tag or index followed by the index of the instance being referenced.
Of course, this representation is much less stream-friendly.
Again, other UBJSON types are omitted for simplicity:
This example serializes the same Appointment instance data presented before.
That's it. Thanks for reading this long issue.