krotik
commented
7 years ago Hi,
5 Gigs of data is quite a bit. I would, in any case, add it in multiple smaller chunks. I am assuming that the data is stored in one or multiple files on disk.
I can think of 3 different ways on how the data could be added:
- Adding it through the command line using the -import argument. I would imagine a script which creates import files and then runs EliasDB's import.
- You could use the REST API. However, for this amount of data it is probably too slow and just the wrong mechanism.
- You could write a Go program which embeds EliasDB. It could read the JSON data in chunks and use transactions to feed it into the datastore. (I think this is the best alternative.)
In general: the power of a graph database lies in its ability to store highly connected data. It sounds as if your data uses primarily keys which might be better stored in a normal key-value store. However, if you do want to store it as a graph (e.g the advancement of the blockchain could be modelled as edges) you would need to form small objects of key value pairs. In EliasDB the keys must be strings while the values can be anything. The value of the "key" and "kind" attribute should also be a string. (It is currently a bug that the "key" attribute value must be a string).
Node values can contain any (and all) JSON data. If you embed EliasDB in an own Go program you can even store slices and object structures. In fact anything which can be serialized with gob.
faddat
It's blockchain data, and there's a key that contains an array, and the keys in that array are almost always different. This blockchain is advancing rapidly, and I can't really say if things will even out too soon (which would enable me to make some kind of struct for this array). So, I noticed the cool trick with the JSON import:
Will this work for any and all JSON?
Say I have 5 gigs of JSON.... what is the best way to feed it to elias?