Sync data from AppEngine to app

[breber]

In order for the app to work well with the AppEngine website, we need to sync data back and forth. This is a challenging problem, due to the fact that records can be changed at different times on different devices, which causes problems when trying to decide which record is "correct" (much like the problem we seem to have with git/svn and merge conflicts).

A little background on what we have so far - in the carhub-auth branch, the project is set up with a git submodule that handles the authentication stuff. Then to perform an authenticated HTTP request, just extend the class AuthenticatedHttpRequest and override a few methods (see one of the existing files for an example). Right now, we just fetch all records for the specific request (maintenance, fuel, vehicle list, etc) from AppEngine and overwrite what we have in the database.

All the records that will be syncing with AppEngine extend from the java class SyncableRecord which has a few properties - mRemoteId, which is the id of the record in the AppEngine datastore, mDirty, which represents whether the record has been modified locally and has not been synced to AppEngine since the modificiation, and mLastUpdated which is the last time the record was modified.

My thought for syncing goes something like this:

1. Fetch all records from AppEngine that have been modified since last sync time
2. Update the local database with the values from this request
   • If a record has mDirty = true and it was modified on AppEngine, just overwrite the local change
   • Set all the records we just updated to have mDirty = false
3. Query the local database for all records that have mDirty = true, and send those to AppEngine
   • Question: For records created on the device, the field mRemoteId will be null - can we just have the handler for sending records to the server respond with the id and update the local database?
   • When the record has been sent to AppEngine with a successful response, set mDirty = false

I think that will work, but I wanted to get everyone's opinion before I try and code the syncing stuff.

Thoughts?

---

Tasks

• [x] Set up the UI to list existing records
• [x] Fetch existing records from AppEngine
• [x] Delete records if they no longer exist on AppEngine
• [x] Create a page to add vehicles
• [x] Create page to add fuel records
• [x] Create page to add maintenance records
• [x] Create page to add base expense records
• [x] Send new records to AppEngine
• [x] Send edited records to AppEngine
• [x] Update AppEngine API to handle fetch changes since a specific time (last modified)
• [x] Create icons for the vehicle page
• [ ] Modify vehicle information (license plates, etc)
• [x] Automatically sync records for a vehicle once they open that vehicle
• [x] Sync data periodically using a SyncAdapter

[jgkujawa]

Sounds ok to me. On Dec 26, 2012 11:56 PM, "Brian Reber" notifications@github.com wrote:

In order for the app to work well with the AppEngine website, we need to sync data back and forth. This is a challenging problem, due to the fact that records can be changed at different times on different devices, which causes problems when trying to decide which record is "correct" (much like the problem we seem to have with git/svn and merge conflicts).

A little background on what we have so far - in the carhub-auth branch, the project is set up with a git submodule that handles the authentication stuff. Then to perform an authenticated HTTP request, just extend the class AuthenticatedHttpRequest and override a few methods (see one of the existing files for an example). Right now, we just fetch all records for the specific request (maintenance, fuel, vehicle list, etc) from AppEngine and overwrite what we have in the database.

All the records that will be syncing with AppEngine extend from the java class SyncableRecord which has a few properties - mRemoteId, which is the id of the record in the AppEngine datastore, mDirty, which represents whether the record has been modified locally and has not been synced to AppEngine since the modificiation, and mLastUpdated which is the last time the record was modified.

My thought for syncing goes something like this:

1. Fetch all records from AppEngine that have been modified since last sync time
2. Update the local database with the values from this request
   • If a record has mDirty = true and it was modified on AppEngine, just overwrite the local change
   • Set all the records we just updated to have mDirty = false
     1. Query the local database for all records that have mDirty = true, and send those to AppEngine
   • Question: For records created on the device, the field mRemoteIdwill be null - can we just have the handler for sending records to the server respond with the id and update the local database?
   • When the record has been sent to AppEngine with a successful response, set mDirty = false

I think that will work, but I wanted to get everyone's opinion before I try and code the syncing stuff.

Thoughts?

— Reply to this email directly or view it on GitHubhttps://github.com/jgkujawa/CarHubMobile/issues/7.

[breber]

One thing I forgot to mention - deletion. Deletion will probably be the most problematic. I don't even know how we will handle that...

[anelson6]

This is a nitpicky suggestion, but rather than overwriting the local change if a record was also modified on AppEngine, should we instead keep the most recent modification? The most recent modification in one spot might even include the previous modification from the other spot if the user chose to re-enter information when they noticed it hadn't changed.

As for deletion, would it be difficult to add a record property such as isDeleted? Then when we set it to True in either location it will show up as recently modified / dirty and we can then delete it for real in both locations after syncing. We would just have to change our HTML pages to not display records with isDeleted set to true, or modify our queries so those records aren't returned and then create a separate query for syncing in which they are returned. So then, the fourth step in your list would be to actually delete all records both locally and on AppEngine with isDeleted equal to true.

[breber]

So in step 2, instead of overwriting always, just keep the one with the latest last modified time? If the one we have locally was modified after the one from AppEngine, leave it marked as dirty and don't update it. I think that makes sense.

I also think the isDeleted property makes sense, though I don't know that is ever safe to delete the record from AppEngine. If a user has multiple devices syncing with the same account, not all of the devices would be notified of the change. So I would say that we update our queries to just not display records that have been deleted, and just actually delete records on the device, and only mark items as deleted on AppEngine, never actually deleting records.

[breber]

So I think one way to deal with deleting records is to add an API method that will list the active records. I just did this for vehicles, and it doesn't appear to have used any of our datastore quota (if it does use quota, it should only use 1 datastore query). So by getting the list of active record IDs, we should be able to tell on the Android app whether a record has been deleted or not.