Implement RESTful document data-store

[scottzer0]

It's clear that augur will need an alternative, CENTRALIZED data store for some of the data on market and possibly comments, etc. This would be an interim solution until decentralized solution mature. This would also be a way to reduce gas costs while augur grows up.

I'm recommending MongoDB, because it fits the bill, is very mature, fast and i'm familiar with it.

The store should accept a doc type and hash id as query and return any meta data seemed fit to supplement on-chain data.

account (much of this will also be stored in the ethereum client's db and check there first)

• list of contacts and associated address
• profile data (blob of avatar, handle, name, etc.)

market

• longer description of the market detailsing specific rules etc.
• image blob

event

• price history of each outcome (time stamp, price, ?)

[tinybike]

I like MongoDB as well, but I thought it would be better in this case if we had a portable datastore (i.e., one that doesn't require installing/running a separate program like MongoDB, Postgres, etc.). This is especially true since this is a localhost app, so concurrency/locking isn't so important!

In fedc66b2a56a9238d3d69171c2a51a38ab5e28c3, I refactored the server-side code to use NeDB as the local datastore. NeDB is an embedded Node database that uses the same syntax as MongoDB. Since it installs via npm, there's no extra overhead from the user's perspective.

Since we'll want efficient asynchronous communication between the client and the local datastore, I also went ahead and switched from node-static to express + socket.io. So we've got a pretty standard (albeit minimal) Node-with-websockets setup.

Right now the datastore is just storing basic user info -- Ethereum account, ether, cash, and reputation balances -- which is sent back to the server from loadAssets. Should be straightforward to store the info you've listed here though -- that's next on the list! Once I've got all that squirreled away in the NeDB, I'll send you a pull request.

[niran]

I thought this was intended as a centralized store for data we're not storing on the blockchain? That is, not a store for the current server.js, but a store for a webservice we'd run to associate with a Clinton prediction market off of the blockchain. Eventually, this sort of thing would move to IPFS/Swarm.

[scottzer0]

yes. a centralized place for all clients to set and get meta data for markets. not a local store. we can lean on the already available key/value store of web3.db (geths leveldb, i think) for local persistence needs.

lets not add complexity and bloat to the client if we can avoid it.

On Apr 23, 2015, at 8:20 AM, Niran Babalola notifications@github.com wrote:

I thought this was intended as a centralized store for data we're not storing on the blockchain? That is, not a store for the current server.js, but a store for a webservice we'd run to associate with a Clinton prediction market off of the blockchain. Eventually, this sort of thing would move to IPFS/Swarm.

— Reply to this email directly or view it on GitHub.

[scottzer0]

also, the node server is purposely a light-weight, static server as an interim gap to host it locally and on heroku for ease of testing. the client should be a bundle that can be delivered and run entirely in a browser and not depend on any other resources except those available by modern web browsers, the ethereum stack and other decentralized stores (ie. swarm, whisper, etc.)

this mongoDB issue, is about giving us an interim tier to host off-chain meta data that ALL clients can access.

meditate on this vision and architecture, mister puffin.

[tinybike]

Oh ok I see. Yeah, in that case concurrency etc. matter. Mongo is probably easiest for this, but since we're dealing with financial stuff, probably should bite the bullet and just use a real RDBMS. Heroku has good Postgres support (and I know you've worked with it before) so I'll just go with that.

[tinybike]

Btw, if we're having users run Node at all, I'd say it's better to use Express instead of node-static, just because it's more flexible (and they have to install something via npm either way).

[scottzer0]

mongo, elastic search, big table are so suited for this task: give them a document hash-key over a RESTful api (builtin with many) and get a structured json data response. nothing in these docs will be sensitive, just less-essential data to flesh out the app. an RDBMS, is not the right tool for this job and would actaully take more engineering to make do the job.

[tinybike]

Yeah I agree document store is the natural model here. But, as far as I know, if you're storing financial data, you have to use an RDBMS. MongoDB isn't ACID compliant.

[joeykrug]

Who cares about whether we're ACID compliant? This is not secure financial data

[tinybike]

At this stage, it isn't. But it will be at some point, right?

[scottzer0]

while Express is a lovely framework for node and does provide static file serving (and a lot more), let's just let this totally working and very straight-forward server be and focus on testing other stuff that's may or may not be working

stop trolling here.

[joeykrug]

It's market metadata which must be hashed & checked client side to make sure it matches the ethereum hash. I should be able to do a data breach and steal all the data (in fact, i should have access to all of it anyway) and it shouldn't matter a single bit

[tinybike]

If we're absolutely 100% sure we will never ever transact locally, then we can use a doc store. But it's easy to use an RDBMS at this point so why not do it? This isn't a security issue in the sense of data access, since as you say it's all public. Rather, it's a data integrity issue. Atomicity in particular is the thing that matters here. Can we get that with Mongo? No, we cannot. So we use an RDBMS.

[scottzer0]

closing this here, cause it's really going to be the responsibility of augur.js how it wants to deal with the metadata not on chain. also, this thread, geez.