orbitdb / field-manual

The Offical User's Guide to OrbitDB
208 stars 43 forks source link

OrbitDB pinning service #83

Open coyotespike opened 6 years ago

coyotespike commented 6 years ago

Because orbitdb is based on js-ipfs, and js-ipfs does not yet have garbage collection, all orbit databases are persistent. In IPFS terms, orbit databases are pinned by default.

However, when run in the browser, an orbit database uses localStorage (and leveldb to cache). This is a surprisingly robust method of storage, but it is still a bit vulnerable. In addition, for a new browser user to sync to the orbit database, another user must be online simultaneously. A user who starts an orbit database and then switches devices will have the same problem.

As haad says, this is the classic bootstrapping problem.

It would be helpful to have an easy way to run a pinning service which will replicate a database and stand ready to sync with new users, to provide resilience and availability. This pinning service would be a cluster of orbit database servers, and so this issue is closely related to orbitdb/orbit-db#165, setting up an orbitdb cluster.

haadcode commented 6 years ago

@coyotespike this is a great idea and something I've been thinking about for long time never having the time to implement. We always needed such "bootstrappers" (always available nodes that can serve the content when others are offline) for Orbit.chat but never had them.

At its simplest, I think, having a way to easily boot up an orbitdb node and start syncing (replicating) a database would be enough to get started, but I really like the idea of a cluster as it gives much more flexibility and power to manage the availability.

I know you're already working on something @coyotespike, so if there's anything we can do to help, let us know and please do share the code if/when possible :)

cristiano-belloni commented 5 years ago

@coyotespike @haadcode in its simplest form, can't a bootstrapper be a node process somewhere in the cloud which just replicates a list of databases? For minimal effort, it could even be a slightly modified version of the Getting Started code running on an Headless Chrome process :) Or I'm missing something?

coyotespike commented 5 years ago

I think that's right! You just have to give your bootstrapper the database address to replicate, or a list of them. Hopefully we will open-source a pinning service soon.

abh1manyu commented 5 years ago

@coyotespike @cristiano-belloni I'm also trying to figure out best way to pin/save DBs from other peers running browser based IPFS nodes. Did you find an optimal approach to handle this? TIA!

cc @haadcode @shamb0t

Jon-Biz commented 5 years ago

Hi all,

In the process of familiarizing myself with orbitdb (and meet my own needs), I've created this pinning service:

https://github.com/Jon-Biz/orbitdb-pinner

You can pass/configure it with an orbitdb address to pin, or have it respond to http requests to /pin and /unpin.

It uses it own orbitdb instance to keep track of the pins, so pins persist between restarts. What's more: you can pass it the address of another orbit-pinner, and have it follow that one, creating a cluster.

aphelionz commented 4 years ago

Moving this to the Field Manual Repo <3