YapDatabase + ReactiveCocoa + CocoaAsyncSocket : KVO take 2 (question)

Hello,

First off I would like to say that you have an excellent library here in YapDatabase. I am using it in another project of mine and I love the simplicity and freedom it provides me over a traditional RDMS.

I have a challenging question/advice to ask you about to see if YapDatabase will work with in my situation. I have model classes that all inherit from MTLModel (Mantle Framework). I have an architecture that fetches remote data from an endpoint (most up to date state data for a user) and deserializes into their respective model classes using Mantle on startup lets say. Also on startup I open a socket to streaming API using CocoaAsyncSocket which parses events and sends NSNotifications out to other parts of the application. Once the data/objects returned my API are in memory they start listening for the posted notifications. The objects filter for the notification and event data they care about and if there is a match then the object updates itself with new data. This keeps the models I just created/fetched the most up to date because a lot of events can be generated during the course of the application being active.

I have a few datasource classes that act as a sort of cache for the fetched data so that they can stay in memory and receive events on the socket (similar to Server Sent Events) and not be dependent on the storage in a view controller (iOS). When the refresh completes and the datasource has new data the controllers are notified using KVO (ReactiveCocoa) and told to reload the collection view. When the collection view renders the cell the collection view cell subviews are then bound (listening via KVO, RAC) to changes on the models properties.

What I have been trying to solve is how to have a better data cache so when the app starts up I can quickly render the last known data. I like what YapDatabase has to offer for views and change notification because another thing I need is the ability to know when something gets added to my list or subtracted so I can monitor the change and animate the UI accordingly. What I am struggling with right know is if my setup above really falls into the pitfalls of using KVO from fetched YapDatabase models. Or if there is a way for me to use YapDatabase and still keep my architecture of update the models with the socket events. Also it seems like I would need a way to re-save to YapDatabase after a socket event has changed the model so that my database has the most up to date information for the next launch.

So maybe the above wasn't so much of a question but more of an advice to how someone might handle a situation like I have described. Maybe it's not possible with YapDatabase, I am fine with that. Currently we are encoding our object graph to disk using NSKeyedArchiver so maybe there is something I can create on top of that to get what I need. More or less I am open to the wonderful suggestions that the smart people here can advise on.

Thanks, Kyle

maybe the above wasn't so much of a question but more of an advice to how someone might handle a situation like I have described

I'll do my best to provide advice.

There are several issues I see with the general architecture. 1) The objects themselves are listening for notifications

This is a problem, because it means that if you have hundreds of objects in memory, then a single NSNotification is going to cause hundreds of method invocations. Imagine 100 instances of the same object class in memory. They have all registered for the same notification. And inside the notification handler, they all perform a string comparison to see if the notification applies to them. Which means that when the notification gets posted, you immediately incur the overhead of 100 method invocations, and 100 string comparisons. All this just to get the notification to a piece of code that can do "the right thing". That's a lot of unnecessary overhead.

2) The objects update themselves

This might sound cool. But, in fact, it's a hazard. Why? Because the object is no longer thread-safe. In fact, are you posting that NSNotification on the main thread? If not, then you're updating your objects on a background thread, while they are simultaneously being used on the main thread. If so, then you're incurring all that overhead mentioned in issue 1 on the main thread, and possibly causing your UI to stutter.

3) "I have a few datasource classes that act as a sort of cache for the fetched data so that they can stay in memory and receive events on the socket"

Do you see how your architecture decisions in 1 & 2 have mandated this extra code? And also require all your objects to be in memory at the same time? This won't scale to thousands & thousands of items. Additionally, it means that you have to load all your objects into memory immediately on app launch. Which is going to cause a delay.

4) When the collection view renders the cell the collection view cell subviews are then bound (listening via KVO, RAC) to changes on the models properties.

Not a great idea. Why? Because its better to do batch updates of your collection view. Imagine updating multiple objects due to a network response. Each change is going to invoke the KVO stuff, causing you to update the collection view. The end result is excess overhead, and the collection view animations might get screwed up. Contrast this with a single batch update, matched with a single update of the collectionView, and a single smooth animation.

how to have a better data cache so when the app starts up I can quickly render the last known data

exactly

another thing I need is the ability to know when something gets added to my list or subtracted so I can monitor the change and animate the UI accordingly

spot on

is a way for me to use YapDatabase and still keep my architecture

Even if the answer was yes, I wouldn't recommend it. Allow me to propose a very different architecture.

You already have a bunch of socket code. Change it so that this code directly updates the objects. Like this:

- (void)socketDidRecieveChanges:(NSArray *)changes
{
    [backgroundConnection asyncReadWriteWithBlock:^(YapDatabaseReadWriteTransaction *transaction){
        for (id changeInfo in changes) {
            // Fetch corresponding object
            id originalObject = [transaction objectForKey:changeInfo.key inCollection:@"things"];

            // Make a copy (as recommended in YapDatabase wiki articles)
            id objectToUpdate = [originalObject copy];
            // Now update the object
            [objectToUpdate updateWithChangeInfo:changeInfo];

            // Save updated object to database
            [transaction setObject:objectToUpdate forKey:changeInfo.key inCollection:@"things"];
        }
    }];
}

You'll notice a few things about this code:

The update code is happening on a background thread, so it's not slowing down your UI
The update code makes copies of your objects, so it's completely thread safe
The update code is saving the changes to disk, so everything is immediately ready on next app launch

So how does your collectionView work now?

You setup a YapDatabaseView, which properly groups & sorts the objects you want to display in your collectionView. And on app launch, that view will immediately be ready for you. So you get super fast app launches. Basically, YapDatabaseView acts as your datasource for your collectionView.

Further, you no longer need the datasource cache stuff. Instead of forcefully pulling everything into memory, you just let YapDatabase automatically handle all the caching for you.

And updates are handled automatically using YapDatabaseModifiedNotification. So when you do that batch update code above, it will post a notification to the main thread. And you can use this information to get a list of all the items that were changed, as well as the items that were added and removed.

I know this is very different from what you have now. But YapDatabase already does the caching stuff for you. And it also does the notification stuff too. I guess the change in thinking is this:

Instead of thinking about individual changes to individual objects, think instead about commits.

As in, when you execute a readWriteTransaction, that's a commit. And every single change you make within that commit, the YapDatabaseView automatically translates it into a single simple list that you can use to properly update your collectionView (with animations).

This might be a bit confusing, and I probably didn't do a great job explaining everything. So I'll link to a few wiki articles that may explain things better.

Cache LongLivedReadTransactions YapDatabaseModifiedNotification Performance Primer Views

I hope this helps. Let me know if you have any questions.

yapstudios / YapDatabase

YapDatabase + ReactiveCocoa + CocoaAsyncSocket : KVO take 2 (question) #80