nothirst / TICoreDataSync

Automatic synchronization for Core Data apps, between any combination of Mac OS X and iOS: Mac to iPhone to iPad to iPod touch and back again.
https://github.com/nothirst/TICoreDataSync/wiki
807 stars 61 forks source link

Whole Store Uploading: Be smarter about handling large store files #29

Closed ghost closed 11 years ago

ghost commented 11 years ago

[update, 2013-01-29] As per the conversation below, uploading large whole store files can be problematic because the user shouldn't be allowed to update the store while the upload is happening. Perhaps we can copy the store off to a temp location before kicking off the upload.

[original message] After downloading whole data store, local device needs to apply all previous changes

When the user uploads the whole data store on device A (say, their iPhone), it uploads a copy of the core data store, and as they make changes it begins to upload change sets. However, when the user downloads the data store onto device B (say, their iPad), it has to first download the initial copy of the whole data store, and then successively apply all of the change sets. This is very time consuming, especially if there is a time lag between when device A first uploaded the data store, and when device B downloads it, because there can be many change sets.

MrRooni commented 11 years ago

This is true, which is why we recommend that devices periodically upload the whole store file so that other devices have fewer change sets to apply when setting up syncing for the first time. You can use the TICDSDocumentSyncManagerDelegate method:

/** Invoked to ask the delegate whether the document sync manager should automatically initiate a Whole Store Upload at registration.

 @param aSyncManager The document sync manager object that sent the message.

 @return `YES` if the sync manager should initiate the upload, otherwise `NO`. */
- (BOOL)documentSyncManagerShouldUploadWholeStoreAfterDocumentRegistration:(TICDSDocumentSyncManager *)documentSyncManager;

to tell the document sync manager that it should upload the whole store.

ghost commented 11 years ago

Yes, but... isn't this a non-ideal solution for users with large core data stores? If you have a 20 MB store (or even a larger one), for example, then it will take a long time to upload -- and you don't want the user to do anything while uploading the datastore, because those changes would not be reflected in sync changes.

MrRooni commented 11 years ago

Uploading large amounts of data can be a problem. There are some steps you can take in the UI to let the user know it's happening, but we can probably make some improvements to the framework here. Off the top of my head I'm thinking that we should copy the persistent store to temp space before starting an upload so that it can happen in the background without affecting user interaction.

ghost commented 11 years ago

I agree that allowing the user to continue using the app while the data store is uploading would be a good thing. But if the user does anything that would make changes to the data store, while it is in the process of uploading—wouldn't that invalidate the data store as the base version of the sync data?

MrRooni commented 11 years ago

Yes and no. The whole store would then be behind the latest changes, but the change set generated from any changes the user makes would make up the difference.

ghost commented 11 years ago

Thanks. Now, if I have the user upload a new version of the whole store, what would happen with the existing sets of sync changes that are stored on the server?

Example: There are two devices, A and B, and the user sets up sync and starts making changes, which are uploaded to Dropbox. Let's say that three months later, device C is connected and the whole store is downloaded as well as all the changes which takes a lot of time, because there have been many changes. However, let's say that just before device C was connected, device A uploaded a new version of the whole store. When device C is connected, it downloads this latest version which includes almost all of the changes--which would be great. However, will TICoreDataSync ignore all of the changes that were made before version 2 of the whole store was uploaded?

MrRooni commented 11 years ago

Yes, TICDS will only download the changes since version 2 of the store was uploaded.

MrRooni commented 11 years ago

TICDS now copies the whole store to a temp location before uploading. Additionally the framework supports compressing the whole store so upload will be faster and use less bandwidth.