mereacre
commented
5 years ago Adds the matrix datatype.
To use, define your field with the __tdxType: ["ndarray"] and addData a numpy.array, ie:
from nqm.iotdatabase.database import Database
import numpy as np
db = Database("", "memory", "w+")
db.createDatabase(schema={
"dataSchema":{
"k": {"__tdxType": ["number"]},
"a": {"__tdxType": ["ndarray"]}, #new ndarray type
},
"uniqueIndex": [{"asc":"k"}]
})
db.addData([
{"k": 0, "a": np.array([1, 2])},
{"k": 1, "a": np.array([3, 4])},
{"k": 2, "a": np.array([[5, 6], [7, 9]])}, # 2d array
])Each ndarray is stored as a JSON string in the dataset, ie:
example = {
"t": "=h", // numpy typestring, h means signed 16-bit int with native alignment
"s": [766, 480], // shape of array, means 766 x 480 (2d)
"v": "f", // metadata version, f means p is a pointer to an uncompressed binary file
"c": arr.strides, // the value of teh strides for the numpy array arr
// filename can be anything, but currently it is being generated by
// vvvvvvvvvvvvv - base64 unix timestamp in ms, this means files are in alphabet chronological order
// vvvvvvvv - pseudorandom prefix to avoid clashes if there is the same timestamp
// vvvv - static suffux
"p": "AAABaBQNuQI=s8ffou_6.dat" // binary file location
// when loading the filename, the path /path/to/database.suffix + .d/ are prepended to it
// ie, if p is example.dat, and db is test.sqlite, we will load test.sqlite.d/example.dat
}@aloisklink Instead of setting ["c"] to true or false it is better to set it to the actual stride value using arr.strides. It will be easier to translate it to javascript.
For example:
X = np.array([[0,1,2],[3,4,5]], dtype='int16')
print(X.strides) # result: (6, 2)
["c"] = X.strides / sizeof(int16)
print(["c"]) # result: (3, 1)
aloisklink
We need to convert from Javascript types:
to Python types.