search

thaliproject / postcardapp

A sample app to demonstrate how to build Thali applications
MIT License
22 stars 5 forks source link

Architecture of Thali based applications #130

Open bwhisp opened 8 years ago

bwhisp commented 8 years ago

Hello,

I am trying to make an application that simply syncs a database between 2 devices (let's say Androids).

I have tested the postcard app, which is quite impressive I would say, but I still have some difficulties in understanding how it works. I would love to have some explanations about the architecture of Thali based apps, especially concerning CouchDB related topics. By the way, in the postcard app, how many instances of CouchDB are running ? I noticed that one is created in jxcore/app.js but I wonder if Thali creates another one for peer to peer syncing purposes.

Also, does Thali automatically connect to peers and replicate their database or do we need to use any API function to "tell Thali" to do it ?

What are the main features performed automatically by Thali and what are the ones we need to "ask" Thali to do ?

Thank you for your consideration and your replies.

PS : I hope you good luck, this is very impressive. And sorry for my english.

yaronyg commented 8 years ago

Just by way of warning, the postcard app is running on an early experimental version of Thali that is really just a prototype. It's not intended to be reliable. We are working on a new release of the Thali code that should be released in late March/early April that should be vastly more stable and we will then move the Postcard app over to that.

The version of Thali that the postcard app is running on is in 'promiscuous mode'. The way this works is that in JXcore we are running a single DB and anytime we discover anyone we will automatically sync our DB with their DB. This is 100% insecure. The point of the prototype was just to test the basic pieces (e.g. can we connect over peer to peer protocols? Can we run node? Can we talk to native code? Etc.). So right now there is nothing to configure because the app will find anyone it can and send them everything it has and take everything they have.

In the upcoming release this will all change. Thali will switch to doing discovery using a secure encrypted framework that protects the identity of both the user who is doing the discovery and the device being discovered. For the gory details on how this works please see http://thaliproject.org/PresenceProtocolForOpportunisticSynching/ and http://thaliproject.org/PresenceProtocolBindings/

But the general model (and yes, we'll be writing an article summarizing this, right now we are heads down coding) is that every user's device generates a public key. Devices have to be provisioned with the public keys they will trust. This can be done with identity exchange (see here for the spec and here for an example with the postcard app). Unfortunately identity exchange WON'T be working in our next release because our first customer doesn't need it. But we'll be bringing it back quickly. Otherwise a central server can be used to provision keys.

Once the Thali plugin has the list of keys two things happen:

Replicate Out - Anytime the local PouchDB instance running in JXcore that Thali has been told to watch changes then the app will try to find all the listed devices that is has keys for to give them the update.

Replicate In - Anytime the device finds another device on its list of keys then it will pull changes from them to its database.

So in this way the device will keep its database in sync with the peers whose keys it has been given. This works over P2P (e.g. bluetooth/ble on Android and multi-peer connectivity framework on iOS) as well as over local WiFi where allowed (many WiFi access points are configured to prevent multi-cast discovery and point to point connections so even when a device is on WiFi it might not be able to use that WiFi to move data and will need to switch to the P2P protocols).

bwhisp commented 8 years ago

Hello, thank you for your quick reply. I will investigate on the links you gave me while waiting for the new release of Thali. If I have a central server with a PouchDB instance and I want to be able to sync with either the server or a peer (when the server is unreachable), do I need 2 PouchDB instances in my app or only one will do the trick ?

yaronyg commented 8 years ago

One will do the trick. That is the power of CouchDB (the protocol that PouchDB implements). It supports full multi-master replication. So you can have a single DB on the device that can be pointed at the cloud, at your peers, anywhere and you will end up with a consistent state or at least have a clear notification of where there are conflicts. Part of the trick of building CouchDB apps is figuring out how to design them so that conflicts are either rare or in many cases, impossible.