XLNT / gnarly

🤙 A blockchain ETL tool to make your life better. Now that’s fuckin’ gnarly.
https://xlnt.co
Apache License 2.0
119 stars 9 forks source link

Add PouchDB Support #32

Open shrugs opened 6 years ago

shrugs commented 6 years ago

PouchDB is a cross-platform database. It's possible to build a database, dump it, and then load it into another PouchDB instance, which is how we'd like to support client-side snapshots (it also supports real-time sync, which is nice for reactive frontends!)

The logic for resuming from a snapshot is:

  1. Drop local databases (reset)
  2. Download the database dump
  3. Load this database dump.
  4. Find the blockHash of the most recent transaction recorded
  5. Restart all reducers from that latestBlockHash

or, when syncing

db.replicate.from(remote)

In this way, a client can download snapshots from trusted third parties (eventually we'll use a decentralized computation network to produce these indexes) and then resume following the index in real-time, locally, block-by-block talking directly to an Ethereum node.

The first step to supporting PouchDB snapshots is to add support to Gnarly's IPersistInterface for talking to PouchDB. By default PouchDB will use an indexeddb adaptor in browser and a leveldb adaptor on node, which is probably fine. If there are any inconsistencies, though, we can use the websql adaptor in browser and the pouchdb-adapter-node-websql adaptor in node (which ends up writing to disk via sqlite).

Use pouchdb-server to persist data serverside.