Datastore and statement metadata

[vyzo]

Closes #36

• Implements the datastore for statement metadata with rocksdb.
• Provide a /data/put and /data/get API endpoints for storing/retrieving metadata objects.
• Implement the /mediachain/node/data protocol for retrieving statement metadata from remote peers.
• Merge statement metadata along with statements in the merge implementation.

Note: There is a minor change in the remote query protocol, as the end/error messages where refactored to be reused in the data protocol.

Datastore API details:

• /data/put accepts a batch of DataObjects, which are json objects with a data field containing the data base-64 encoded, and returns a stream of object keys (which are B58-encoded SHA256 multihashes)
• /data/get/{objectId} retrieves a data object by its key or 404s.

Example:

$ head -c 20 /dev/urandom | base64
FCG389RpiSmWkbK96fd0if4gmhw=
$ curl -s -H "Content-Type: application/json" -d '{"data": "FCG389RpiSmWkbK96fd0if4gmhw="}' http://127.0.0.1:9002/data/put
QmT7KTeJYJ7pnvsUk8g56AqFgp1Cqk91HnuhyhmaYZW4dr
$ curl http://127.0.0.1:9002/data/get/QmT7KTeJYJ7pnvsUk8g56AqFgp1Cqk91HnuhyhmaYZW4dr
{"data":"FCG389RpiSmWkbK96fd0if4gmhw="}

Merge API changes:

• /merge will now fetch metadata referenced by the statements in the result sets, if it doesn't already have them in the datastore. The API returns two integer values (instead of just one), the count of statements and objects that were merged. Missing metadata in the peer are treated as errors and abort the merge.

Example:

# the dataset contains a bunch of statements which all reference the same object
$ curl -H "Content-Type: application/text" -d 'SELECT * FROM * LIMIT 10' http://127.0.0.1:9004/merge/QmW5vgizcxphSSV22fKxxNCnv9TbaidYDiBCbHR9bGg2RM
10
1
$ curl -H "Content-Type: application/text" -d 'SELECT * FROM * LIMIT 10' http://127.0.0.1:9004/merge/QmW5vgizcxphSSV22fKxxNCnv9TbaidYDiBCbHR9bGg2RM
0
0
$ curl -H "Content-Type: application/text" -d 'SELECT * FROM * LIMIT 100' http://127.0.0.1:9004/merge/QmW5vgizcxphSSV22fKxxNCnv9TbaidYDiBCbHR9bGg2RM
90
0

[parkan]

🎆