Rich-text collaboration

[dfahlander]

Dexie Cloud already support syncing text properties, but we lack specific conflict-free CRDT support for collaborative text editing of same document.

We have a demand from customer to add support for realtime communication of text editors using tiptap or similar components. For implementation, we should see what we can use from Tiptap and Hocuspocus-server to integrate it seamlessly in dexie cloud.

[dusty-phillips]

That would be such a killer feature.

[dfahlander]

I'm looking into integrating Y.js into Dexie.js, not only Dexie Cloud. So this issue will be twofold:

Dexie.js ❤️ Y.js

The plan is that Dexie.js will support Y.Docs as properties on objects and thus be a storage provider of Y.js in itself even without any addon required. If it turns out to be too much of added code (which I do not think it will be), we might lift this out to an addon instead, but since Y.js ecosystem is so promising and leading, it feels more right to support it natively.

Dexie's role in the Y.js ecosystem will be similary to y-indexeddb but allowing docs to be organised into props of objects and queried just like other props of Dexie tables. Allow tagging docs and index them and reference them from other tables. Also the lifetime of the Y.Doc instances will be cached and handled, taking away the burden for developers to maintain and organize Y.Doc instances.

Dexie Cloud ❤️ Y.js

Dexie Cloud will handle Y.Docs seamlessly and sync, support realtime editing of text using its existing connection and access control features. Dexie Cloud will also support Y.js awareness protocol for seeing other user's cursors etc in text documents.

Since Dexie Cloud supports partial replication and shared data collaboration, Y documents becomes a natural sharable data type and collaboration will respect the permissions set for the realm of the object it resides in.

[dfahlander]

Update

Dexie + Y.js support is almost done. Kevin (Y.js author) has been very helpful with actively supporting me in API questions. The dexie-cloud support is still to be implemented but the design for it is done.

[scottrblock]

This is excellent @dfahlander! Will it eventually be possible to use Dexie Cloud REST API to update a Y.js doc? My use case would be from a node.js or similar server/endpoint.

Tangential, but I am assuming REST API is best way to access/update records in Dexie Cloud from a backend because the npm package would rely on indexdb being available, but not sure.

[dfahlander]

I think we'd need to collect the requirements on REST API for sync use cases together to find a solution. The /sync endpoint is possible to call from a server in theory but there are lots of undocumented data formats etc that would need to be documented. Supposely we could add a simplified sync api that is targeted for node clients rather than dexie-cloud-addon on web.

Adding a REST endpoint for Y-updates would not be a problem though, just want to think it through to get it right.

[scflode]

Is there anything one can help with the PRs (#2016 & #2045)? Yjs support as mentioned before would be awesome!

[dfahlander]

Thanks! The Y.js support for both dexie and dexie-cloud is becoming ready for test in just a few days but I will need help trying it out in real apps. #2045 is based on 2016 and is the PR I've been working on.

There's already an alpha dexie@4.0.9-alpha.1 on npm that makes it possible to store Y.Doc instances in dexie and I'll be releasing a new alpha of dexie, dexie-react-hooks and dexie-cloud-addon within some days where awareness and sync are implemented.

I took a preliminary decision to let dexie support Y.js documents "natively" and not requiring an addon for it. I might change my mind on this but for now it is "built-in" into dexie. The caller need to provide the Y library in the constructor options to Dexie since I don't want to introduce new dependencies. Had I implemented this as an addon, yjs would probably be a dependency of the addon instead.

How dexie stores Y.Docs

In contrast to y-indexeddb who only stores a single document in a dynamically created object store, Dexie treats Y.Doc instances similar to any other type such as string, Blob, Date etc so that one can still think in terms of tables and objects, where every object can hold one or several properties with documents in them. In the background though, every update to a document is stored in a dedicated table tied to the table of the parent object and the property it is stored in. In order to create these tables, users need to declare which properties might hold Y.Docs, similar to how primary keys and indices are declared.

Declaration

import Dexie from 'dexie'
import * as Y from 'yjs';

const db = new Dexie('friendsWithNotes', { Y });

db.version(1).stores({
  friends: `
    id,
    name,
    age,
    notes:Y` // ... ":Y" declares 'notes' as a property that contains Y.Doc instance
});

In dexie@4.0.9-alpha.1, documents are loaded by creating a provider as such:

import { DexieYProvider } from 'dexie';
import { db } from './db';

// Query a row:
const friend = await db.friends.get(1);
// Get the Y.Doc instance (it'll always be there for all friends):
const yDoc = friend.notes;
// Start loading the document:
const provider = new DexieYProvider(yDoc);
// Wait until loaded:
await yDoc.whenLoaded;

// And when done with it:
yDoc.destroy(); // ...will also destroy provider

In the upcoming version, I've added a reference counting mechanism so providers can be kept alive through re-renders and be shared between components loading the same document.

const provider = DexieYProvider.load(yDoc);

And when done with it:

provider.release(); // decrements a refcount instead and when zero, destroys both doc and provider.

dexie-react-hooks

A new version of dexie-react-hooks (also released in a few days) has a useDocument() hook to use in place of DexieYProvider.

import { useLiveQuery, useDocument } from 'dexie-react-hooks';

interface Props {
  friendId: number;
}

function MyComponent({friendId}: Props) {
  // Query a row
  const friend = useLiveQuery(
    () => db.friends.get(friendId),
    [ friendId ]
  );
  // Load the Y.Doc that resides in friend.notes
  const provider = useDocument(friend?.notes); // ok to pass undefined here (on first render)
  if (!provider) return null; // still loading the friend
  // At this point:
  //   provider.doc holds the 'notes' document (friend.notes === provider.doc is true)
  //   provider.awareness will be present if dexie-cloud-addon is used.
  // Here it's ok to pass the provider further to text editor components that support doc and awareness.
}

I'll send a comment on this thread when all things are ready to try and all helps in testing it and giving feedback and ideas are welcome.

[dfahlander]

Update: Everything is now in place for using Y.js with Dexie and Dexie Cloud. Anyone interested to try the alpha, please contact me or post on this issue. What you need is:

• Ask me to run a script that migrates your dexie.cloud database and puts it into staging.
• Install dexie@next, dexie-cloud-addon@next and dexie-react-hooks@next (if using react)

Plain Dexie.js users can try the dexie@next alpha with the previous instruction on this issue.

A sample app is also currently being built that will demonstrate TipTap with Dexie, Dexie Cloud and Y.js.