search

signalapp / Flock

Private contact and calendar sync for Android.
https://signal.org/blog/flock
359 stars 80 forks source link

Discussion: Alternatives to DAV as storage #93

Open untitaker opened 9 years ago

untitaker commented 9 years ago

Cont. https://github.com/WhisperSystems/Flock/issues/75#issuecomment-90160667

I am trying to avoid derailing the original issue. Please continue conversation here.

untitaker commented 9 years ago

I'd like to know from @rhodey which aspects were taken into consideration when opting for WebDAV as a storage backend. I imagine it must have been

OTOH there are a lot of downsides to CalDAV/CardDAV:

I could go on...

This is supposed to be a more theoretical/far-future thread about possible alternatives to WebDAV that Flock could support.

@rhodey mentioned remoteStorage as a possible backend. The spec does look nice and minimal in every way, is easy to implement too. However, it is not battle-tested, and doesn't currently fulfill the two criteria mentioned above.

Since Flock is already based on (parts of) a file-protocol, it is natural to look after alternative file-protocols:

gusennan commented 9 years ago

@untitaker Nice thread. Just an observation, it seems that a lot of developers are going the dropbox route with their synchronization needs. One of the challenges, I think, with choosing any of the above options would be the necessity of creating a file-organization protocol on top of whatever transportation mechanism one chooses. With CardDAV and CalDAV, the way that the organization of calendars and contacts is clearer, though standardization. If it were taken that the organization of data is something worth addressing, then, it could also be possible to include just standard web-apis (Restful APIs, perhaps) for example.

untitaker commented 9 years ago

I think, with choosing any of the above options would be the necessity of creating a file-organization protocol on top of whatever transportation mechanism one chooses.

CalDAV/CardDAV themselves are already defining directory structures for their items (since they are based on a file protocol themselves). I already reuse this structure for local storage.

If it were taken that the organization of data is something worth addressing, then, it could also be possible to include just standard web-apis

Sure, but that would require writing your own server as far as I'd know. The point is that we choose something that the user likely has installed.

Another route one could go is the decentralized one, but I strongly doubt anybody has the resources for that.

gusennan commented 9 years ago

@untitaker I agree with both points. With FTP, for example, would it not be necessary to define one's own directly structure?

untitaker commented 9 years ago

Yes, but I don't think that's much of a problem.

gusennan commented 9 years ago

I've heard good things about using Couchbase and Couchbase-lite to sync between mobile clients.

rhodey commented 9 years ago

@untitaker at first glance WebDAV seemed to have a bunch of OSS libraries, clients, and servers-- for these reasons I felt at the time it would be both less work and more friendly to the OSS community to use WebDAV. months later now, I'm of the opinion that most OSS WebDAV libraries, clients, and servers are not fit for production, this partially because WebDAV as a protocol is not fit for production in 2015, in the 90s and early 2000s it was probably fine.

the only existing alternative I can recommend at this time is remote storage

untitaker commented 9 years ago

Any comments on using a remote FS as storage engine?

nysatrok commented 9 years ago

Would it be reasonable to store the data in local iCal and vCard files and encrypt these files, so we can sync them with an arbitrary protocol?

nickel-chrome commented 8 years ago

@rhodey and @untitaker have you looked at the Weave Sync protocol (aka Firefox Sync)?

https://docs.services.mozilla.com/storage/apis-1.5.html

I was trying to solve the same problem a couple of years ago, i.e. end-to-end encrypted contacts and calendar sync, and went through a similar process of evaluating what was out there. In the end I decided that built-in encryption support and key exchange were the key problems and settled on a Weave Sync with a Firefox Account (FxA) backend.

https://github.com/nickel-chrome/CucumberSync/wiki/White-Paper

I have found Weave Sync's REST based protocol to be well designed and quite straightforward to implement. I've recently finished a second cut of a prototype Android app, CucumberSync, which supports the new Firefox Accounts backend, which although harder to implement than Weaves Sync provides an solid, if not elegant, solution to password based key exchange, including changing and resetting passwords, without requiring the server to be trusted.

https://github.com/nickel-chrome/CucumberSync/releases/tag/v1.1-alpha-2

untitaker commented 8 years ago

How much does this rely on FxA? As far as I know setting up a fully self-hosted server is unreasonably hard, if not intentionally.

On 31 July 2015 14:25:54 CEST, nickel-chrome notifications@github.com wrote:

@rhodey and @untitaker have you looked at the Weave Sync protocol (aka Firefox Sync)?

https://docs.services.mozilla.com/storage/apis-1.5.html

I was trying to solve the same problem a couple of years ago, i.e. end-to-end encrypted contacts and calendar sync, and went through a similar process of evaluating what was out there. In the end I decided that built-in encryption support and key exchange were the key problems and settled on a Weave Sync with a Firefox Account (FxA) backend.

https://github.com/nickel-chrome/CucumberSync/wiki/White-Paper

I have found Weave Sync's REST based protocol to be well designed and quite straightforward to implement. I've recently finished a second cut of a prototype Android app, CucumberSync, which supports the new Firefox Accounts backend, which although harder to implement than Weaves Sync provides an solid, if not elegant, solution to password based key exchange, including changing and resetting passwords, without requiring the server to be trusted.

Reply to this email directly or view it on GitHub: https://github.com/WhisperSystems/Flock/issues/93#issuecomment-126674697

Sent from my phone. Please excuse my brevity.

nickel-chrome commented 8 years ago

The Weave Sync protocol itself doesn't strictly rely on FxA at all. It's broken in to two parts, Server (aka Storage API) and Client (aka Global Storage Format). The current version of Firefox Sync uses Storage API v1.5 for which mozilla have released an implementation aimed specifically at self-hosting that is straightforward to setup provided you are comfortable with python.

https://github.com/mozilla-services/syncserver

Naturally out of the box the v1.5 sync server is designed to work with FxA, i.e. HawkAuth/BrowserID based authentication. In theory it would not be hard to change the backend auth provider, but in that case I would simply use the v1.1 protocol, the key difference of which is authentication, it uses Basic Auth and thare are a number of OSS implementations.

https://github.com/owncloud/mozilla_sync

https://github.com/posativ/weave-minimal

I've written a java client and modified an existing python client that works with both v1.1 and v1.5 currents if you want to experiment.

https://github.com/nickel-chrome/weaveclient-java

https://github.com/nickel-chrome/weaveclient-python

untitaker commented 8 years ago

Right, there's the old sync protocol and the new one. The old one (the one implemented by weave-minimal) seems to be fine AFAIK, but I meant the new one. Last time I checked, ownCloud had problems implementing it and @posativ didn't support the new one because of the concerns mentioned.

There's also the question of how easy it is to run a server. I recall a Bugzilla ticket where I got the impression that Mozilla is not actively supporting self-hosting at the moment. This used to be better with the old protocol, though the old reference implementation was awfully slow.

I don't think that Mozilla is ready to promote that protocol as a standard yet. I'd like to see reusable (maybe Firefox Sync-like, I don't know) encryption and key exchange on top of remoteStorage.

On 3 August 2015 13:10:06 CEST, nickel-chrome notifications@github.com wrote:

The Weave Sync protocol itself doesn't strictly rely on FxA at all. It's broken in to two parts, Server (aka Storage API) and Client (aka Global Storage Format). The current version of Firefox Sync uses Storage API v1.5 for which mozilla have released an implementation aimed specifically at self-hosting that is straightforward to setup provided you are comfortable with python.

https://github.com/mozilla-services/syncserver

Naturally out of the box the v1.5 sync server is designed to work with FxA, i.e. HawkAuth/BrowserID based authentication. In theory it would not be hard to change the backend auth provider, but in that case I would simply use the v1.1 protocol, the key difference of which is authentication, it uses Basic Auth and thare are a number of OSS implementations.

https://github.com/owncloud/mozilla_sync

https://github.com/posativ/weave-minimal

I've written a java client and modified an existing python client that works with both v1.1 and v1.5 currents if you want to experiment.

https://github.com/nickel-chrome/weaveclient-java

https://github.com/nickel-chrome/weaveclient-python

Reply to this email directly or view it on GitHub: https://github.com/WhisperSystems/Flock/issues/93#issuecomment-127197061

Sent from my phone. Please excuse my brevity.

nickel-chrome commented 8 years ago

I like the idea of implementing a storage agnostic solution, e.g. remoteStorage, but I've not found you gain much by doing so. In my experience the hard work is predominantly in the client and in particular key-exchange. I was also originally hoping to find a key-exchange solution that would work over the top of a storage agnostic backend, but in practice I found this unwieldy and, without careful implementation, quite possibly insecure. In summary to support secure key-exchange between client devices requires a one time key and multiple round trip messages, as I learnt first hand while experimenting with roll your own key-exchange solution.

https://wiki.mozilla.org/Services/Sync/P2P_Key_Exchange_And_Rotation

Having tried a number of key-exchange methods for Firefox Sync, Mozilla has put together a good summary of conventional key-exchange methods between devices. They are also looking to make another attempt to provide a more secure key-exchange option to compliment the convenience of the onepw protocol.

https://wiki.mozilla.org/Services/Sync1_5/FxASyncPairing

In the end the constraints and difficulties of implementing device based key-exchange brought me back to FxA, which takes care of key-exchange and changing passwords while maintaining a reasonable level of security, i.e. they store only the equivalent of a hashed and salted password and importantly the password does not leave the device.

https://github.com/mozilla/fxa-auth-server/wiki/onepw-protocol

I also looked at Flock key-exchange, but IIUC when using a third party hosted WebDAV server a hostile systems admin could brute-force the sync key with the same difficulty as standard hash based password. Hence to achieve the same level of security as the Flock hosted WebDAV server you would need to self-host (and possibly modify Flock), hence I found myself back looking at dedicated key-exchange/auth solutions.

FxA is the best password based key-exchange/auth solution I've yet to come across (although Flock is also good) hence while investigating alternatives to WebDAV I'd recommend having another look at Mozilla's self-hosted sync server v1.5, which has improved significantly in the last year. I am self-hosting at cucumbersync.com and was able to set it up without any major difficulties. It requires more effort than weave-minimal, but is not difficult if you are familiar with python. Note that the latest versions of Firefox now support configuration of a self-hosted sync server. Feel free to use my server to try it out. The token server url is:

https://cucumbersync.com/syncserver/token/1.0/sync/1.5

It shouldn't be too hard to port the token server/auth functionality of Mozilla's v1.5 implementation to weave-minimal as they are both written in python, however there would be more work required to implement v1.5 in the ownCloud mozilla_sync application, which would need to implement the token server/auth functionality from scratch.

One important caveat regarding FxA is that self-hosting an FxA account server is non-trivial so if hosting the account management was important this would need to be taken in to account. I'm going to look in to this myself in order to provide a more seamless signup process for CucumberSync. I'd also really like to provide a web portal the synced data, which I'm hoping may soon be a reasonably secure proposition given Google's e2e library and hopefully similar advances in Firefox and Safari.

@rhodey do you have any thoughts on key-exchange/auth (and ideally changing passwords) for Flock? Am I correct about the security issues when using third party hosted WebDAV servers? And finally are the security constraints of third-party hosting satisfied by remoteStorage?

untitaker commented 8 years ago

I don't understand why you refer to remoteStorage as a storage-agnostic backend.

nickel-chrome commented 8 years ago

Apologies my mistake, I was working on a reading of the summary page. I thought remoteStorage acted as a front end to Amazon S3, Google Drive etc.

I guess my last question is most relevant, i.e. does remoteStorage help solve key-exchange/authentication for end-to-end encryption?

Also, I don't have easy access to a computer just now so sorry for drawn out discussion.

untitaker commented 8 years ago

As far as I understand it doesn't. I've never come to fully read the spec but as far as I remember, it seems like a simple data storage with some sort of permission handling (so apps have to request access to a particular storage).

On Thu, Aug 13, 2015 at 02:56:14PM -0700, nickel-chrome wrote:

Apologies my mistake, I was working on a reading of the summary page. I thought remoteStorage acted as a front end to Amazon S3, Google Drive etc.

I guess my last question is most relevant, i.e. does remoteStorage help solve key-exchange/authentication for end-to-end encryption?

Also, I don't have easy access to a computer just now so sorry for drawn out discussion.

Reply to this email directly or view it on GitHub: https://github.com/WhisperSystems/Flock/issues/93#issuecomment-130860481