Evaluate text app HPB sync options

[juliushaertl]

Initial Experiments

• [x] Try plugging in the new collaboration extension into nextcloud text -> works if we drop user colors
• [x] Check how easily we can use https://tiptap.dev/hocuspocus/ as a drop in replacement for our polling based sync backend
  • [x] First implementation uses y.js steps but sends them over the existing sync backend and still uses client side logic to save the file

Drop in Replacement

https://github.com/nextcloud/text/issues/2847

Use the yjs based collaboration extension with the existing server API (+small modifications). Turn the sync service provider experiment into a working solution.

• [x] send and store steps encoded as base64
• [x] combine multiple yjs changes into a single change before they are send to the server
• [ ] remove the DB lock when pushing changes to the server
• [x] write tests for joining an existing editing session and collaborating
• [x] write tests for opening an existing md file and collaborating on it
• [x] test performance under load - maybe during contributor week.

Explore combining with other providers

During company week:

• [ ] See if using notify_push just for the message passing is an option that may improve the situation https://github.com/nextcloud/text/issues/1805
• [ ] Try combination of sync service provider with webrtc
• [ ] Try combination of sync service provide with hocuspocus
• [ ] Prioritize / pick combination to work on (first)

Combine with Hocuspocus

Use Hocuspocus to send steps and awareness messages between clients more efficiently.

• [ ] Allow configuring a hocuspocus url
• [ ] Define a mapping from a text file to a hocuspocus document
• [ ] Implement access control like tokens or JWT to validate that clients are allowed to connect to the web socket of a specific document.
• [ ] We need to clarify how this could be handled from a license/release perspective as hocus-pocus is currently in private beta, repos is private, npm package is public already

Awareness (user presence and authorship)

• [ ] Use yjs awareness instead of our current user session tracking
• [ ] explore implementing user colors based on yjs awareness
• [ ] plug in user cursor extension

Fully integrate

Combine the various providers seamlessly and make use of their strength.

• [ ] Serialize markdown and store it to the server from within hocuspocus
• [ ] Detect available providers in the client and use them accordingly
• [ ] Compress document history

[max-nextcloud]

We also need to look into authentication - both of the clients connecting to the hokuspokus server as well as the auth of the write from the hokuspokus server to storage backends - especially if those are special backends such as shares or external storages.

[juliushaertl]

As discussed, let's focus on the following in order:

• plugging in hocuspocus as as an optional component for syncing the steps
  • first make general collaboration work
  • save process
  • still save the document on the client
  • Concurrent saving could be avoided by having one client that pre-announces the save process through web socket so that others won't save in the meantime
  • authentication
  • maybe use JWT with a shared secret to verify access of sessions
• Explore options to use y.js with our existing php backend

[juliushaertl]

https://discuss.yjs.dev/t/stateless-server-broadcasting-implementation-in-go/393 Is an interesting read for the option to reuse our php backend as a fallback with y.js, so this should work while having no compression of the previous steps (which we don't have anyways with the current approach) so y.js would even be more lightweight by not using json for the steps.

[max-nextcloud]

https://github.com/nextcloud/text/tree/experiment/yjs has a minimal working tiptap2 collaboration based on tiptaps public websocket server.

[max-nextcloud]

https://github.com/nextcloud/text/tree/experiment/minimal-yjs keeps more of our current session handling.