Phase 3: Sync Engine and Real-Time Collaboration

[youknowriad]

This is the overview issue for the collaborative editing project as defined on this post.

The goals are multiple here:

• Allow multiple users connected to the same WP-Admin to collaborate together on the same content (multiple posts/templates...) live.
• Enabling offline editing and synchronization of data.
• Offer a great DevX where developers are freed from thinking about the collaborative editing needs. Data is collaborative and synced by default and abstracted away from the UI developer.

Tasks:

• [x] Review previews explorations (#1930 and asblocks).
• [x] Bootstrap the sync engine architecture and package (being experimental flag) #52681
• [x] Allow persisting and loading data from a local database. (browser, offline-first data)
• [x] Build a transport communication layer between multiple peers working on the same documents. (WebRTC based on long polling or poling?) #53922
• [ ] Extensibility: Allow the transport layer to be replaced ?
• [ ] Design and represent the “presence” of connected peers in the interface. Integrate selection state with other useful parts of the interface, like the block list sidebar.
• [ ] Implement a security layer on top of the transport layer to prevent private data from being shared between peers without the right permissions.
• [ ] Lift concurrent editing restrictions. Right now WordPress locks a post to the current user that is editing it. Another user can take over, but it doesn’t allow two users working on the same post at the same time.
• [ ] Review undo / redo stacks and resolution.
• [ ] Implement the syncing behavior for all the entities of WordPress Data (CRDT representation and config).
• [ ] Employ peer caret and selection primitives that work across block types and design their semantic and visual representation.
• [ ] Explore the possibility of “following a user” as they work through a document.
• [ ] Assess performance and memory usage and impact on non-collaborative editing sessions.
• [x] Support multi-site (both multi WP and single WP installs) local databases (document ids) https://github.com/WordPress/gutenberg/pull/52681#discussion_r1265498194
• [x] Support syncing for global styles edits https://github.com/WordPress/gutenberg/pull/52681#pullrequestreview-1532982849
• [ ] Check that all entities are synced properly.
• [ ] Implement e2e tests for collaborative editing and offline syncing.
• [ ] Support inline comments
• [ ] Allow peers to make edits as suggestions to later be accepted/rejected.

[prajapatisagar]

Hello Team,

As a first task of this issue, I have added my findings on https://github.com/WordPress/gutenberg/pull/23129 PR.

Thanks

[youknowriad]

Hi @prajapatisagar Thanks for the review there. Actually, the exploration has moved now to this PR https://github.com/WordPress/gutenberg/pull/52681 That was a good reminder though, I'm going to close that old PR now.

[youknowriad]

I merged the initial bootstrap behind an experimental flag and added a number of todo list items to the current issue. You all feel free to own one of these tasks and iterate on it.

[annezazu]

Out of the list of subtasks for real time collaboration, I've created an issue for inline block commenting so there's a more dedicated space to discuss and explore: https://github.com/WordPress/gutenberg/issues/59445

[creativecoder]

I found this presentation very informative, there may be some useful information for the collaborative sync engine:

"New algorithms for collaborative text editing" by Martin Kleppmann (Strange Loop 2023)

Specific points to highlight:

Automerge is a CRDT (conflict-free replicated data type) engine with a variety of API libraries available (including JS!). Of particular interest it the Peritext merge algorithm which is designed to handle issues specific to merging richtext/markup. I could see this being very useful for creating more elegant revisions for block content.

https://github.com/WordPress/gutenberg/assets/1699996/5219ef2e-636d-4968-b84b-69aac765938e

As an example using that technology, there's a prototype collaborative text editor called Upwelling, which is kind of like a combination of Google Docs and git. There's likely some design and technical inspiration there, particularly for asynchronous collaborative editing.

[annezazu]

For anyone following this work, I wanted to connect the dots to this discussion issue from @dmonad. Kevin is sponsored by Automattic to move this work forward and happens to be the author of Yjs! Two weeks ago he shared a lengthy update complete with video demos of some of what he's been working on.