[Data sync] Make data sync less error prone and easier to maintain

[gino-m]

The current data sync workflow is complicated, making it hard to understand and prone to errors (#2726, #2377, #2751, #2684). This is arguably most critical part of the system, so improving this should be P0+ priority.

The main components in question are LocalMutationSyncWorker, MediaUploadWorker, and SyncStatusViewModel and related fragment and repositories.

Suggestions:

• [ ] Expose a repository method to return mutations grouped by a new UploadQueueEntry model object, grouping create LOI+submission mutations together, sorted in reverse chronological order (oldest first, FIFO).
• [ ] Update LocalMutationSyncWorker as follows:
  • [ ] Use the above repo method to look at the entire queue on each run rather than just for one LOI.
  • [ ] Look at all incomplete mutations, not just FAILED. This will capture stuck operations. Since there will now only be at most one LocalMutationSyncWorker running at a time, this should be safe.
  • [ ] Treat each Operation as an atomic batch, let mutations in the operation fail or succeed together.
  • [ ] If any operation failed, retry() the worker.
• [ ] In addition to triggering LocalMutationSyncWorker on "submit", also trigger periodically (15 min?) to catch any "stuck" mutations, ie in case the worker crashed before it could retry() for some reason.
• [ ] Stop using "delete+insert" instead of "update" throughout and enable cascading on all foreign keys to guarantee consistency of local db and prevent accidental deletion of data.
• [ ] Ensure there's adequate unit test coverage for main use cases as well as all edge cases.
• [ ] Update the sync status screen to use the new UploadQueueEntrys returned by the repo.
• [ ] Simplify Mutation.SyncStatus and split off UploadQueueEntry.Status

See also https://github.com/google/ground-android/issues/2235

@shobhitagarwal1612 @scolsen @sufyanAbbasi @jabramowitz5 @lecrabe @jo-spek @kenstershiro I would suggest prioritizing this over all other feature requests. Let's discuss.

[gino-m]

@shobhitagarwal1612 Let's clean split this work up and make this airtight.

[shobhitagarwal1612]

Should we also add a manual button to trigger data sync?

[gino-m]

Should we also add a manual button to trigger data sync?

The current behavior is to automatically retry whenever a connection is available, and I plan to add logic to periodically check the queue in case the last sync crashed. In that case, what purpose would the button serve? My concern is that by adding a button users will think they need to click it to have data be uploaded.