fangq
commented
7 years ago @SOTelemarkForever, I have been working on a JSON-based general data format - JData specification - for a little while but never completed. The goal is to develop a general format to encapsulate arbitrary form of hierarchical complex data structures using JSON/UBJSON. The syntax in encoding a high-dimensional array, complex arrays, sparse arrays etc in JSONlab are largely influenced by my thoughts in this partially drafted specification:
http://iso2mesh.sourceforge.net/cgi-bin/index.cgi?jsonlab/Doc/JData
I can see the matlab/python form are more natural in JSON syntax, but when processing such data, one can not pre-allocate the array size until one reads the entire array data stream. This is not necessary an efficient approach. The embedded array construct [[],[],...] also does not imply grid-like structure because you may have [[1,2], [3,4,5],[6, 7]]. So, the shape and dimensions are unpredictable.
The form used in JSONlab (as well as the modified UBJSON), the keyword approach eliminates such ambiguity and allow a user to define classes, dimensions and other characteristics before processing the actual data stream. This can be useful for dealing with large among of data on a regular basis.
Eventually, when I complete the specification, I should also provide some simple libraries for different language to convert such data structure to a regular array. Such conversion for matlab already exists in JSONlab
https://github.com/fangq/jsonlab/blob/master/struct2jdata.m
I will close this issue but feel free to continue discussions in this thread.
As I'm using data back and forth between Python and Matlab/Octave I'm wondering if you're considering to change the behavior the coding of Arrays larger then 2 dimensions. I can always go mental hacking away and make my own version but there are probably more people around doing the same interchange
Comparing jsonlab with for example Matlab2016 there is a incompatibility in the JSON data.
jsonlab encoding: "psi": {"ArrayType": "double","ArraySize": [3,3,2],"ArrayData": [0,3,6,1,4,7,2,5,8,5,8,11,6,9,12,7,10,13]}
compared to Matlab2016/Python encoding: "vz": [[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11, 12]]]
Complex numbers and sparse Matrixes/Arryays are not supported in Matlab2016/Python and is usually encoded in two sets of data e.g. x_real, x_imag (complex number) and expand the sparse matrix before JSONing it