Generic Sync:investigate sync abstraction

[cse-test-sync]

This largely is implementing the algorithm/algorithms for sync and conflict resolution described in the remerge spec.

It also requires implementing vector clocks and the like, if that's not done as part of local storage.

┆Issue is synchronized with this Jira Story ┆Story Points: 3 ┆Epic: chrome.sync.storage (Generic Sync) prototype ┆Sprint: SYNC - end 2019-12-20

[linabutler]

@thomcc and I talked about how we could integrate Remerge into Desktop last week. Conceptually, it's pretty similar to how the Desktop machinery works today: we need a way to handle resetting sync IDs, feeding incoming records into the store, and uploading outgoing records. The additional meta-collections mean we have to pass those records around, too, but I don't think that complicates things.

Things we can and can't use on Desktop:

• We can use rusqlite as-is, since we'll also need it for rkv. Investigating how to vendor rusqlite so that it links against the version of SQLite in m-c, instead of its own, is on my to-do list.
• We can't depend on sync15, because m-c vendors all Rust crates, including (AIUI) optional dependencies. This means depending on sync15 will pull in rc_crypto, viaduct, and everything else.
• We can use serde, and the other dependencies mentioned in Cargo.toml. Most of them are already vendored, too!
• We can, and should, continue to use blocking I/O in the Rust component. In m-c, there's a new API for dispatching background tasks specifically for blocking I/O.

One way to approach the integration is something like bug 1596322 suggests, and pass cleartext JSON strings back and forth. We'd add a skeleton, subclassable RemergeEngine to JS (it'll look a lot like BufferedBookmarksEngine, and add a Rust XPCOM component that takes care of updating the mirror, merging, and fetching staged records for upload.

For the storage part, we can provide a CRUD-style API that also takes JSON strings. (We could reflect it as an XPCOM interface instead, but using interfaces for simple structs like NativeRecord is super kludgey). On the JS side, we'll provide a wrapper to make this more ergonomic.

Aside: If you're thinking, hey, this sounds a lot like ffi_support JSON, you're right! XPCOM is an FFI, with additional memory and type safety guarantees. XPCOM methods can take JS objects directly, via in jsval parameters, but the Rust XPCOM bindings don't support those.

So the integration would look something like:

• Make a remerge_sync15 (remerge_rc?) crate that depends on remerge and sync15. This is what we'll use for Fenix and other consumers. This crate will implement sync15::Store, which take a sync15::IncomingChangeset and deserializes each sync15::Payload to the corresponding Remerge type. Outgoing schemas and records will likewise get serialized from a Remerge type to a sync15::Payload, bundled up in a sync15::OutgoingChangeset, and uploaded.
• In m-c, make an in-tree crate that depends on remerge directly. This will implement an XPCOM interface that takes JSON strings, and deserializes them to a remerge::RawSchema or remerge::RawRecord on the Rust side. After the merge is done, we'll serialize Remerge records to JSON, pass them back to JS via an XPCOM callback, and let the JS Sync implementation encrypt (as-is, without any additional parsing or serializing) and upload them.
• Remerge will define its own storage APIs for applying deserialized records, and fetching serialized ones for upload. This API will also need to handle pulling in records from multiple collections—the Remerge collection itself, and its companion meta collection with the schemas and client info.

Our glue crate is also going to need to handle:

• Sync ID changes, for both collections. We can, and should, make them identical, and raise an error if the meta collection's sync ID ever gets out of step with the main collection.
• Getting and setting the last sync time.
• Signaling completed and failed uploads, from JS Sync to Remerge.

I'll close this issue in favor of the Bugzilla bug, since we'll use that for the prototype, and don't want to fork the discussion. This comment is specifically so we can document the plan in Jira/GH for tracking.