Make Flexible Sync Less Painful By Auto-Opting Types Into Syncing

[bdkjones]

Problem

The new(ish) Flexible Sync feature is...complicated. I understand why it exists, but it's:

1) Unintuitive 2) Very messy to set up — nested closures as a function parameter in flexibleSyncConfiguration with braces vomited all over the place. 3) Very easy to mess up — when I add a new type to the model, I have to remember to go add it to my subscriptions and there's no compiler help to remind me. 4) If my newly-added type links to other types, I have to remember to add those types (and manually check if I already did add them) or I get the worst possible outcome: a silent failure with no error messages and just nil objects at the end of links when, in fact, those objects aren't nil in the database. (That ALONE should have been disqualifying for shipping this implementation of Flexible Sync because it's essentially silent data corruption, but that ship has apparently sailed and we're stuck with what Mongo did.)

The old Partition Based sync was much cleaner. Especially for cases where developers just want to sync everything.

Solution

Because Partition Based sync is pretty clearly deprecated, I'd like the Swift SDK to offer a much cleaner API to deal with Flexible Sync. This API does one thing: "Sync everything in my database, except these things: X, Y, Z"

If I don't provide a type to exclude, this new API just sets up all the subscriptions I need for every object type I have. That way, I never need to worry about false-dead-ends in linked objects or manually adding a subscription when I create a new model object—it restores Realm to how it worked before this Flexible Sync API asteroid hit.

In short, this new API would be an "opt out" setup rather than an "opt in" setup. That's much more appropriate for the kinds of apps I build with Realm and neatly solves the problems above.

Note: I don't use queries in subscriptions because there are no queries for me to use. I need all the objects of every type in my app. So I just have a list of subscriptions with object types—a list I now need to manually update and track. I understand that the query approach makes sense for many apps, but not in my case.

Alternatives

Partition Based sync, I guess? The website has big warning labels on it though, so I'm sure it's not long for the world.

How important is this improvement for you?

Would be a major improvement

Feature would mainly be used with

Atlas Device Sync

[dianaafanador3]

Hi @bdkjones thanks for taking the time for giving some feedback about flexible sync. We're still tuning the API to offer a better developer experience while maintaining speed and performance.
We are working on an extension of the flexible sync API which soon we will release as a preview https://github.com/realm/realm-swift/pull/8244, I know this doesn't solve your problem, we'll love some feedback when this is released as well.

[bdkjones]

@dianaafanador3 Thanks. #8244 doesn't look like it'll help too much. It's become a little irrelevant, I guess, because I've just about had it with Flexible Sync. I spun up a new Atlas App Service, turned on Flexible Sync with developer mode enabled, ensured that the default Role was set to always allow read/write access and when I try to do this very simple thing:

try! realm.write({
    realm.add(newFoo)
})

I get this garbage:

It's a configuration issue, I'm sure. But the backend of MongoDB is such a blackhole of pane after pane after pane after pane of pain, I have no idea. I'm not querying a field. The role permissions are set. I opened the realm like this:

let config = app.currentUser!.flexibleSyncConfiguration { subs in
            subs.append(QuerySubscription<Foo>(name: "foo"))
        }

Realm.asyncOpen(configuration: config, callbackQueue: .main) { result in
    ...
}

I'm tired and I'm frustrated and I want to throw MongoDB out and start all over with a different service. I'm guessing that the "feedback period" that Mongo ran on Flexible Sync last year didn't really include a broad swath of real-world developers because we're all very busy and can't take production apps to new, untested APIs. Mongo didn't get a lot of honest feedback about this thing and, as such, it shipped a very complex, fragile, cumbersome solution to a pretty simple problem: "Let developers tell us what subset of their data on the server they want to sync to devices."

I guess my final question is: Is Partition-Based sync staying around long-term? Because it's way, way better.

[tgoyne]

We are planning on phasing out partition-based sync on the server, and sometime in the not too distant future orgs which don't already have a PBS app won't be able to create new ones.

With recent versions of Realm you can connect to a flexible sync server app using the partition-based API and we automatically generate the required subscriptions. This was intended as a migrational thing, but I suppose it isn't limited to that.

I'm in full agreement that the APIs we have now aren't great for app-first development where you don't already have a backend with way more data than is reasonable to ship to each client. Some of the relevant people are currently out on holiday, but when they return we can hopefully talk about ways to improve this.

[tgoyne]

Untested, but this should automatically subscribe to all objects for all types in your schema:

func subscribeToAll(_ subscriptions: RLMSyncSubscriptionSet) {
    let schema = RLMSchema.shared().objectSchema
    // This check only works if you never remove a type from your schema
    if subscriptions.count == schema.count {
        return
    }

    for objectSchema in schema {
        let name = objectSchema.className
        if subscriptions.subscription(withName: name) == nil {
            subscriptions.addSubscription(withClassName: name, subscriptionName: name,
                                          predicate: NSPredicate(value: true))
        }
    }
}

let config = user.flexibleSyncConfiguration(initialSubscriptions: ObjectiveCSupport.convert(object: subscribeToAll), rerunOnOpen: true)

It turns out our Swift API doesn't really support this sort of dynamic usage due to only exposing typed wrappers of the functions, so this has to drop down to the obj-c API.

[duncangroenewald]

I'll certainly second providing a simpler - SYNC ALL - config option. Our app also needs to just sync all objects.

@tgoyne - now that I have solved my data loading problem with js I am updating the RealmSwift client app - what is the correct query syntax for syncing all objects ?

subs.append( QuerySubscription<AppPermission> { ???? Query to sync ALL OBJECTS ??? })

[tgoyne]

QuerySubscription<AppPermission>(where: "TRUEPREDICATE") will subscribe to all objects of that type.

[duncangroenewald]

@tgoyne of course I knew that lol !

Thanks. It seems the sync download progress might have been fixed now so will look to migrate to flexible sync over the next 6 months.