Dropbox Datastore is Deprecated

[andygeers]

We all need to find new solutions now that Dropbox have announced they are deprecating their Datastore service: https://blogs.dropbox.com/developers/2015/04/deprecating-the-sync-and-datastore-apis/

As I evaluate the different options, I thought it would be helpful to put it somewhere public where people could comment on it and we could keep it up-to-date as helpful suggestions come in.

Here are some of the alternative options:

Option 1) Rewrite ParcelKit using Core API

As a community, we could rewrite ParcelKit to use the Core Dropbox API, aiming to keep ParcelKit's API itself roughly unchanged. One could imagine using a Dropbox folder per Datastore "table", and one JSON file per "record" (using the record ID as the filename).

Pros:

• to the end user, it carries on working much as before - they still use the same Dropbox account as before
• Payment also remains the user's concern - it uses their storage quota and the user pays
• Non-JSON content (i.e. files) can be stored alongside the JSON data
• User data is encrypted

Cons:

• Similar cons to the Dropbox Datastore itself - relative to other options, not all that many people have Dropbox accounts
• Might require quote a lot of work to move to a totally different paradigm.

Challenges:

• Dropbox have told me that the Core API does not support offline access in the way that the Datastore API used to
• You would have to implement a way to handle merge conflicts - you'd lose the elegant way that the Datastore used to do it

  Option 2) CouchBase Lite

See their blog post here: http://blog.couchbase.com/syncing-with-core-data

Pros:

• Very similar model to ParcelKit / Datastore - you can drop in a plugin that makes your Core Data migrate nicely

Cons:

• You have to run the open source CouchBase on your own server
• You have to pay a lot if you want them to do support your CouchBase server

  Option 3) Google Drive Application Data folder

Google Drive's API supports something called the "Application Data" folder: https://developers.google.com/drive/web/appdata

This would be akin to option 1 - just using Google Drive to store JSON files rather than Dropbox.

Pros:

• Lots of people have Google Drive accounts
• Uses the user's own storage quota / the user pays for it.
• Unlike Dropbox Core API, it seems like the Google Drive API supports offline access: https://developers.google.com/drive/android/files

Cons:

• Might be even more work than migrating to the Dropbox Core API
• There's a pretty tiny data limit on how much you can store in Google Drive's "Application Data" folder

Challenges:

• How well does it handle merge conflicts?

  Option 4) Amazon Cognito

As I understand it, Amazon has a roughly equivalent service to the Datastore: "Cognito" http://aws.amazon.com/cognito/

Pros:

• Handles offline access nicely
• roughly similar schema model to Datastore
• most people have an amazon account

Cons:

• There's a pretty tiny number of records permitted, making this essentially useless.
• The developer has to pay all costs which scale according to the number of users and the amount of data they store.
• This gets even worse if you support files such as photos alongside the JSON data (presumably you'd have to store that in your own S3 bucket?)

  Option 5) Firebase / Parse

There are various third-party services that handle sync for you, top ones that come to mind are Parse and Firebase.

Roughly similar models to Amazon Cognito, except that I imagine you have to handle your own user registrations? (Anybody know?)

Pros:

• Multi-platform
• Both handle offline access nicely
• Parse has a nice online admin dashboard
• Parse has a large developer community

Cons:

• Significant rewrite of your application from a ParcelKit world
• The developer pays all costs
• Where do you store non-JSON data such as photos?
• Parse doesn't encrypt user data automatically

  Option 6) CloudKit

For some people, CloudKit will be a suitable alternative, to sync their iOS-only Core Data database.

Pros:

• Better than iCloud Core Data Sync used to be
• All iOS users have an account?

Cons:

• iOS only. For anybody with a cross-platform app to support this is a non-starter.

  Anything else?

There's probably a million alternatives I've missed. Any suggestions?

I'm seriously considering undertaking Option 1 myself, because I have no money to pay for services such as Firebase / Parse! But I have a feeling it might turn out to be harder than I thought.

[daviddelmonte]

I was using Parse before switching back to DB. Parse stores user data in open clear text. As a developer, I could see anything that any user had ever posted. Datastore seemed to protect both users and developers. I really liked using ParcelKit /Dropbox with my app. I appreciate the hard work you have put into it.

Does iCloud core data have any more cross platform capability than CloudKit?

[andygeers]

I'm pretty sure iCloud Core Data thing is to be avoided for reliability reasons - but no, I don't think it's any more cross-platform than CloudKit

[daviddelmonte]

I definitely agree on the first part. The complaints / problems raised in the Apple Support communities on this speak for themselves. Is there a solid reason why Dropbox is stopping datastores? Could they be approached to reconsider?

On Apr 27, 2015, at 10:46 AM, Andy Geers notifications@github.com wrote:

I'm pretty sure iCloud Core Data thing is to be avoided for reliability reasons - but no, I don't think it's any more cross-platform than CloudKit

— Reply to this email directly or view it on GitHub https://github.com/overcommitted/ParcelKit/issues/34#issuecomment-96681770.

[andygeers]

@daviddelmonte I think it's because so few developers were actually using it. And it's unlikely that hundreds of thousands of developers are suddenly going to start building on top of a service that's already been announced as dead!

[paulmorriss]

I haven't used either, but there are a couple of options for Android: http://developer.android.com/training/cloudsync/index.html

[meoz]

During my search for a sync framework that could handle being fully functional offline and sync in the background when online I landed on Parse. Why Parse? Because it's one of the few frameworks that is very transparent towards developers and is actively developed where new features are added all the time. Moreover, cost is not an issue because you have unlimited free requests and would never pay a penny unless you'd have a very successful product (where you'd have other options anyway). I had a look at ParcelKit too but not being multi-user was a dealbreaker.

Pros for Parse:

• a nice online admin dashboard
• large developer community
• there are already opensource frameworks which do a great job already (https://github.com/flavionegrao/APIncrementalStore)
• multi platform
• multi user
• no lock-in, export data functionality
• easy to setup and maintain

Cons:

• no data encryption out of the box

[chaitanyamannem]

Hi andy, i am with you on option 1 or option2, i can help with coding also.

[semerda]

This is a shame. Is Dropbox listening to the developer community to shut down the Datastore? It feels like the Datastore was such a great feature that was unique to their offering. Now they are moving towards another BaaS API in the cloud like the guy next door.

For those only targeting iOS then I believe CloudKit is the best way forward. Anyone have good references where it shows how to setup CloudKit with Core Data sync? Most of the stuff online is pre iOS8 and filled with complexity.

[smarx]

@semerda We're definitely listening. :-)

Of course our team is also full of people who like the Datastore API, but see https://blogs.dropbox.com/developers/2015/04/deprecating-the-sync-and-datastore-apis/ for some of the reasoning behind it. If anyone wants help with anything relating to the deprecation, please write in to our API support team: https://www.dropbox.com/developers/contact. We want to help as much as possible.

[chaitanyamannem]

@smarx we dont want any help, it's mistake to invest our precious time on dropbox, thank you for teaching a good lesson, no more dropbox, i am switching to google drive api. Thank you.

[andygeers]

@smarx Thank you for listening - I know that many within Dropbox were as sad to see the Datastore go as we were. Do you have any feedback on the pros / cons for Option 1 listed above? Am I correct that the offline support would be an issue?

[smarx]

@andygeers Sorry for the slow reply. I like option 1. Offline support is a bit of a challenge, but it's probably not that bad. Offline support means you need to be able to keep track of which pieces of data have changed locally.

The scheme you suggest of a file representing a single key/value pair makes conflict resolution pretty easy... you'll need to pick a strategy and act accordingly. The most likely choice is "server wins," which means when there are conflicting changes to the same file, you choose the server's version.

A very useful mechanism in the Core API to support this sort of thing is that files each have a rev (short for "revision"). When you retrieve a file from Dropbox, you'll get it's rev as well, which you'll need to keep track of. When you update a file, you can specify a parent_rev, which means the upload will be conditional: if the rev has changed, the upload will fail. (In the "server wins" conflict resolution scheme, this would be when you download the new version from the server and discard the local changes.)

Given the ability to listen for changes (via /longpoll_delta and /delta), the ability to keep track of revisions of files, and the ability to perform conditional uploads based on revisions, I think you can replicate much of what the Datastore API gave you by using the Core API.

I hope that helps! Please let me know if there's more I can help with.

[semerda]

@smarx so the majority of Dropbox developers didn't want the Datastore API? if Dropbox is listening then how about showing the developer community some love with an open source project built ontop of the new API which brings back the old Datastore like functionality? or detailed documentation how to achieve this? Just throwing out some options :)

[smarx]

@semerda No promises, but this is definitely on my list of things I want to do. We have a lot of work to do to finish up and ship API v2 first, but I hope to put some open source code out there down the road.

[andygeers]

So, my current thinking is to go with Couchbase Lite: http://blog.couchbase.com/syncing-with-core-data

[andygeers]

In case it's of interest to anyone, I've been modifying ParcelKit to sync to Couchbase Lite's sync gateway instead of Dropbox. I'm about 80% of the way towards getting it working in a dev environment - seems like a really nice alternative to Datastore. Here's my branch: https://github.com/andygeers/ParcelKit/tree/couchbase

[daviddelmonte]

Bravo Andy

On Apr 6, 2016, at 9:31 AM, Andy Geers notifications@github.com wrote:

In case it's of interest to anyone, I've been modifying ParcelKit to sync to Couchbase Lite's sync gateway instead of Dropbox. I'm about 80% of the way towards getting it working in a dev environment - seems like a really nice alternative to Datastore. Here's my branch: https://github.com/andygeers/ParcelKit/tree/couchbase https://github.com/andygeers/ParcelKit/tree/couchbase — You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/overcommitted/ParcelKit/issues/34#issuecomment-206374507

[CaptainMalReynolds]

Not a very useful comment here, but just wanted to say it's pretty sad how Dropbox abandoned Datastore SDK, I was using it for a few years already in my app and was pretty satisfied with it. And now I'm out of a good sync solution. Thanks to the author of ParcelKit anyway, appreciate your time and energy investing into this library.

[andygeers]

It's taken me a while, but I've now ported this project to work against Firebase instead of Dropbox Datastore. I still haven't updated the example project which probably means this isn't going to be very helpful for anybody other than me just yet. https://github.com/andygeers/FirebaseParcelKit