ipfs / notes

IPFS Collaborative Notebook for Research
MIT License
400 stars 33 forks source link

IPNS+IPFS as a NoSQL database #70

Open rethore opened 8 years ago

rethore commented 8 years ago

It would be quite nice if we could have a library that would emulate a NoSQL database. So in essence offering an abstraction of IPNS+IPFS through methods such as insert, update, remove and find. From there we could easily build distributed web apps based on this as a back-end. So for instance meteor is using a combination of a client side minimongo library that syncs with a server side MongoDB. If we were to swap the MongoDB with this new library, and move most of the logic on the client side, we could achieve quite a bit already. Going further would require to run the server side as well locally, but it's not impossible, just less convenient. There is of course some problems arising of having an open access back-end database, but as long as people are not able to erase other peoples entries, their activity, if judge negative by the community, could be filtered out in a p2p fashion.

jbenet commented 8 years ago

@rethore yep, this is very possible. there are many systems that implement NoSQL dbs and even full SQL dbs on top of KV-stores. we should look at them and try them out.

one good route would be to start with things from the level nodejs ecosystem. there's a few SQL adaptors wrapping level.

some interesting links:

whyrusleeping commented 8 years ago

we should take a look at using the files api to build such a thing.

jbshirk commented 8 years ago

https://github.com/bigchaindb/bigchaindb https://www.bigchaindb.com/features/ https://www.bigchaindb.com/whitepaper/ ipfs mentioned

fazo96 commented 8 years ago

The problem is handling permissions. As far as writing and deleting it can all be implemented, but we need a system where each folder has different permissions. It will be possible when we can have more people allowed to publish a given IPNS record :+1:

Kubuxu commented 8 years ago

Can a directory with subdirectories being /ipns/Qm.... be created?

whyrusleeping commented 8 years ago

@fazo96 btw, you can generate a key with this and use it to publish an ipns entry anytime without the daemon with this

This behaviour will be integrated into ipfs itself soon, as well as a mechanism for storing and managing keys. But those two programs should serve as a decent way to accomplish a few things now

fazo96 commented 8 years ago

Wow that's very nice @whyrusleeping! I hope to see that in the HTTP API soon

rethore commented 8 years ago

thanks @jbshirk, I wasn't aware of this project. It does seem to complement IPFS and ethereum quite well. It would be nice to build a distribution stack that bundle those three projects together and integrate them nicely. Ideally the incentivization scheme should look at compensating for bandwidth, CPU and storage. A complete decentralised crypto-economy and internet stack. This is a really exciting prospects.

r14c commented 7 years ago

I'm in the early stages of developing a graph database on IPFS. The idea is to publish metadata (graphs, vertices, edges, schemas) via IPNS as RWORSets of IPFS hashes. There are some performance considerations, but I'm thinking that GPG could do well as the backend to a cryptographic RBAC system.

I'm in a bit of a time crunch getting funding and such, but I'm excited to finally get cracking on Infinite Automata and collaborating with y'all starting this winter.

fdietze commented 7 years ago

@tokyo-jesus I'd like to know more details of your project. Like how would a db schema look like? Can there be something like transactions? How to avoid race conditions?

r14c commented 7 years ago

Schemas consist of primitive types and named references to other schemas. CRDTs don't have a concept of locking, the state is only guaranteed to be eventually-consistent, which is enough for most needs, but not all. Long term, we'd like to enable strongly-consistent transactions through another distributed mechanism that can provide locks.

jbenet commented 7 years ago

@tokyo-jesus would be great! 👍

jbshirk commented 6 years ago

relevant: https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/